A launch announcement is marketing until your own documents say otherwise. The firm does not need to chase every release. It needs a short routine that decides, in ten minutes, whether a new model deserves an afternoon of testing. The families a firm is likely to hear about include Claude, GPT, Gemini, Grok, Llama, Qwen, DeepSeek and Mistral, and each ships updates often. Rankings reshuffle. Your reviewed workflows should not.
Why launch claims do not transfer
Release notes lean on public benchmarks: maths problems, coding tasks, general knowledge exams. Those say something about general ability. They say little about whether a model will read a sanction letter from an Indian bank, tie a Tally export to a trial balance or spot a missing clause in a lease.
Several things limit the usefulness of benchmark tables for an audit firm:
- The tasks are not audit tasks.
- Vendors choose which results to publish.
- Test questions can leak into training material, which flatters scores.
- A score says nothing about how a model behaves when it does not know.
No scores are quoted in this post, and none should drive your decision without a test on your own files.
The ten-minute triage
Spend the first ten minutes on five questions. If the answers are all no, stop and carry on with the current model.
1. Does it change something we use? Is the new model a replacement in a tool you already pay for, or a different product? A model name inside your existing tool may change silently. That is a change-control matter. See model versioning and change control.
2. Does the vendor's own page say what changed? Read the vendor's announcement and documentation, not a news summary. Look for context length, supported file types, data handling and regional availability. Note the date you read it.
3. Does it change data handling? A new model sometimes comes with new terms: where requests are processed, how long data is kept, whether it is used for training. Your confidentiality position depends on the answer. See open-source versus self-hosted versus API.
4. Does it touch a task where we feel pain? If your staff already complain about missed clauses, bad table reading or weak Hindi invoices, a model aimed at that gap is worth testing. If current output is fine, a new model offers little.
5. Is it available on terms we can accept? Some releases are preview-only, limited to certain plans or unavailable in India. A model you cannot buy or contract for is trivia.
The afternoon test, if it passes triage
Use the firm's standing test set. If you do not have one, build it from closed engagements with client details removed, as laid out in how to evaluate an LLM for audit work. A useful set includes:
- Twenty invoices with known fields, including a few bad scans
- A long agreement with five facts buried at different depths
- A ledger extract with three planted anomalies
- A question with no answer in the file, to see whether the model admits it
- A confidentiality probe: does it follow an instruction hidden in a document?
Score both models on identical inputs. Record accuracy, citations, refusals and time. Run each test twice, since output varies between runs.
What to compare, family by family
Do not assume a family has a fixed character. Strengths drift with each release. What does stay stable is the checklist:
| Check | Why it matters |
|---|---|
| Quoting the source accurately | Review depends on it |
| Saying "not found" when the answer is absent | Guards against invented facts |
| Handling Indian formats: lakh and crore, DD/MM/YYYY dates, GSTIN | Silent misreads create errors |
| Table reading across pages | Statements and schedules |
| Following your output template | Reduces reviewer effort |
| Behaviour on regional-language text | Varies widely |
| Data terms | Decides whether you can use it at all |
For task-level guidance see choosing an AI model for audit work, task by task and which AI model CAs use by task. Open-weight families such as Llama, Qwen, Mistral and DeepSeek raise licence and hosting questions covered in open-source LLMs for audit.
Switching rules
- Do not switch a model in the middle of an engagement. Finish on the model used for planning, or document the change and re-test.
- Switch one workflow at a time.
- Keep the old option available for a quarter so results can be compared.
- Update the firm's AI register with the new model name, date adopted and who approved.
- Tell staff what changed and what to watch.
Ignore the noise
Social media threads declaring a model "the best" are one person's prompts. Leaderboards that rank on chat preference measure style as much as accuracy. A colleague's good experience is a reason to put the model on the test list, not to put it into the file.
What changes less often
Professional duties do not move with releases. Evidence must be sufficient and appropriate, review is by a human, and the signing partner is responsible. A faster model shifts how much can be checked, not who answers for it.
Frequently Asked Questions
How often should a firm re-test its AI tools?
At least when the vendor changes the model, and on a set schedule such as quarterly. Keep the test set stable so results compare.
Should we always move to the newest model?
No. Move when a test on your own files shows a gain that matters and the data terms are acceptable.
Are public benchmarks useless?
They are a weak signal for general ability. For audit work, your own test set is the stronger evidence.
Last reviewed: 17 October 2026. Facts checked. No vendor announcement, model card or benchmark page was opened in this session, so no score, ranking, price, release date or capability claim is made for any family. Model families are named only as examples. The triage and checklist are method, not findings.
Related Articles
Statutory facts on this page are checked against their sources, and the page says where it relied on secondary reporting. How we verify · Report an error