LLM evaluation means testing a model on your own tasks, with known right answers, and counting how often it is wrong. A public leaderboard cannot do that for you. A short written test plan, run before use and again after every change, gives you both a decision and audit evidence.
Last reviewed: 17 October 2026. Facts checked. We ran a web search on LLM evaluation practice (golden datasets, regression testing, LLM-as-judge) and read the summaries of several practitioner guides; these are blogs and vendor guides, not standards, so the numbers below are rules of thumb. We opened OWASP's LLM Top 10 page and NIST's pages on the AI Risk Management Framework and the Generative AI Profile (AI 600-1). We did not open the ICAI standards themselves.
What you are testing, in plain words
You want to know three things.
- Is it accurate enough for this task?
- Does it fail safely? When it is wrong, can a reviewer spot it?
- Is it stable? Will it give the same quality next month?
A model is not good or bad in general. It is good or bad for a task.
Step 1: Define the task and the pass mark
Write one sentence: "The model will extract invoice number, date, amount and GST from vendor invoices." Then write the pass mark: "At most one wrong field in fifty, and every miss must be visible to the reviewer." Set the mark before you see results.
Step 2: Build a test set
- Use real files with identifiers removed.
- Include easy, typical and nasty cases: scans, odd formats, missing pages.
- The guidance we found suggests about 50 to 100 cases, with some portion adversarial or edge cases. Again, a rule of thumb.
- Keep a separate set you never use for tuning prompts. That is your final exam.
- Write the correct answer for each case yourself, before running the model.
Step 3: Run and score
| What to score | How |
|---|---|
| Correct | Matches your answer |
| Wrong | Confident but incorrect |
| Missing | Left out |
| Invented | Not in the source at all |
| Reviewer time | Minutes to check each output |
Invented content is the most dangerous. See AI hallucinations in audit.
Step 4: Use judges carefully
Some teams use a second model to grade the first. The guides we found say this can track human judgement at lower cost, but it has biases and drifts. They suggest re-scoring a small sample by hand and checking agreement. For audit work, we would keep a person as the final scorer on anything that reaches a working paper.
Step 5: Test the failure paths
OWASP's 2025 list of LLM risks includes prompt injection, sensitive information disclosure, excessive agency and misinformation. Its prompt injection page suggests adversarial testing, treating the model as an untrusted user. Try these on your own tool:
- A document with hidden text saying "ignore your instructions".
- A request to show another client's data.
- A question the source documents cannot answer. Does it say so, or make something up?
Step 6: Decide and record
Write a one-page result:
- Task, model, exact version, settings and prompt.
- Test set size and date.
- Scores by category.
- Pass or fail against your mark.
- Controls that apply, such as 100% human review.
- Who approved use.
NIST describes its AI Risk Management Framework as voluntary and gives a Generative AI Profile (AI 600-1) as a companion. They are useful vocabulary for a firm that wants a recognised structure for this record.
Step 7: Re-test on every change
Re-run the same set when the model version, the prompt or the connected documents change. Compare to the last result, and name the cases that got worse. Our post on model change control covers this in detail.
SA 500 on audit evidence asks the auditor to consider the reliability of information used as evidence. A search of ICAI-hosted study material described reliability as including accuracy and completeness. Your test record is how you show that you considered both for AI output. Check the standard's wording before citing it.
Ready-made help
The AI audit tool evaluation checklist gives a structure. The AI governance internal audit workpaper helps document it.
Frequently Asked Questions
What is LLM evaluation?
Testing a language model on defined tasks with known answers, then measuring errors. For a firm, it means testing on your own anonymised files.
Can we trust public benchmark scores?
They tell you little about your files. Use them at most to shortlist, then test yourself. See our task-by-task model guide.
How often should we re-test?
On any change to model version, prompt or source documents, and on a regular schedule.
Who should do the testing?
A senior person who knows the task, not only the person who built the tool.
Is one test enough?
No. Tests show behaviour on that set. Keep human review in place after the test passes.
Related Articles
Statutory facts on this page are checked against their sources, and the page says where it relied on secondary reporting. How we verify · Report an error