Benchmark scores are not your scores: build your own evaluation set to choose an AI model — and how many questions it needs

One week in mid-October 2026: Google says Gemini 4 Argon tops the Vals leaderboard. Anthropic reports Claude Haiku 5.5 beats GPT-6 Luna on every test with results for both. Alibaba says Qwen3.8-27B matches models many times its size on coding tests. All of it may be true — on the tests they chose.

What you need to know is simpler: which model correctly answers the most questions your customers and staff actually ask? No public leaderboard can answer that. This article shows how to answer it yourself — and works out how many questions you need before the answer can be trusted.

Why public leaderboards are not enough

One — they measure other people's work. A composite leaderboard combines tests of maths, coding and specialist knowledge. Your work might be answering customers in Vietnamese about the return policy, or extracting an amount from an invoice photographed on a phone. A model that excels at olympiad maths is not necessarily better at that.

Two — the leaderboards disagree. In the comparison of three models at the same $2/$10 price, Gemini 4 Argon tops Vals but sits five points behind Claude Opus 5.5 on Artificial Analysis. Neither is wrong — they measure different things.

Three — vendor-published numbers are selected. Each lab publishes the tests its model does well on, usually at the highest reasoning level — the most expensive setting, which not everyone uses in production.

Four — the questions may have leaked into training data. Public benchmarks have sat on the internet for years. A model may have "seen" the questions during training, so a high score reflects memory more than ability. Questions taken from your own internal logs have certainly never been seen by any model.

Building an evaluation set from real data

Take questions from logs, do not invent them. Invented questions tend to be cleaner and clearer than real ones. Real ones have typos, abbreviations, two requests in one message, photos taken at an angle. Pull them from the last month or two: customer messages, emails, documents to summarise, questions staff send to support.

Group by task type and take enough of each. For a shop, for example: price and stock questions, returns, complaints, orders, out-of-scope questions. If 80% of messages are price questions, the evaluation set still needs enough complaints — because that is where a wrong answer costs most.

Include three kinds of hard question. Cases the current system has handled wrongly. Ambiguous cases where even a good staff member would ask for clarification. And a few questions where the correct answer is "no information available" — to find out which model will say it does not know instead of inventing something.

Write the answers before running anything. For questions with a clear answer (message group, order code, amount), record it so a script can compare. For open questions (drafting a reply), write three or four specific criteria: does it give the right order code, does it apologise when the customer complains, does it promise something the company does not do. Write them before seeing any model's answer — otherwise the criteria will unintentionally lean towards the first answer you read.

How many questions

This is the part few people calculate, and the reason many internal comparisons reach the wrong conclusion. We simulated the following: two models, one correct 80% of the time and one correct 86% across all the questions it might face — a 6-point gap, roughly the distance between second and ninth place on the current Vals leaderboard. The two models tend to fail on the same hard questions and succeed on the same easy ones, as in practice. Draw n questions at random, grade both, see which scores higher. Repeat twenty thousand times:

• 30 questions: the better model wins 72% of the time, ties 13%, and the worse model wins 15%.
• 50 questions: wins 81%, worse model wins 11%.
• 100 questions: wins 91%, worse model wins 5%.
• 200 questions: wins 98%.
• 300 questions: wins over 99%.

In other words, with a 30-question set, roughly one comparison in seven picks the worse model without you knowing.

The good news: when the gap is large, few questions are enough. With two models at 60% and 85%, 30 questions pick the better one more than 99% of the time. So:

• About 30 questions: enough to rule out clearly weaker models, catch systematic problems (wrong format, failing on regional Vietnamese), and compare costs.
• About 100 questions: enough to separate models about 6 points apart, nine times out of ten.
• About 200 or more: when two models are close and the decision matters — for example choosing the model that will process contracts every day for a year.

Another quick calculation: with n questions and accuracy around 80%, the margin of error on the measured score (95% confidence interval) is about ± 1.96 × √(0.8 × 0.2 ÷ n). With 30 questions that is ±14 percentage points; with 100, ±8; with 300, ±4.5. When two models differ by less than this, do not conclude that either is better.

Same model, two runs, two different scores

Language models do not answer identically every time. On most questions they are reliably right or reliably wrong; but some questions are "borderline" — right this time, wrong the next.

A second simulation: a 30-question set where the model always gets 21 right, always gets 3 wrong, and the other 6 are a coin toss. The average score is 80%. But rerunning the same set on the same model, the score ranges from 73% to 87% in 95% of runs, and roughly one pair of runs in seven differs by 10 points or more.

The consequence: if you run each model once and compare, part of the difference may be pure chance. The simple fix: run each question three times and record its pass rate. A question the model passes only 2 out of 3 times is valuable information — in a real product, that is a question where a third of customers get the wrong answer.

Using AI to grade AI

Hand-grading 200 open answers per model is a lot of work. The common approach is to use a strong model as the grader. It works, but it has documented biases. Research by Zheng and colleagues, presented at NeurIPS 2023 while building the MT-Bench evaluation, describes three:

• Position bias: when comparing two answers, an AI grader tends to prefer the one in a particular position (first or second, depending on the model) regardless of content.
• Verbosity bias: longer answers tend to be scored higher, even when they are not more correct.
• Self-preference: a grading model tends to rate its own answers, or those of models from the same family, more highly.

The same research found that, used carefully, agreement between a strong AI grader and human graders is comparable to agreement between two humans. Three ways to reduce the bias:

• Swap the order: grade each pair twice with positions reversed. Count a win only if it wins both times; if the result changes with position, count it as a tie.
• Hide names and use a grader from a different family: do not let lab A's model grade lab A's answers.
• Check against humans: take a random sample of about 30 questions, have an independent person grade them, and compare with the machine grader. If they disagree a lot, fix the grading criteria before trusting the results for the other 200.

For questions with a clear answer, do not use AI grading — compare directly in code: exact match, regular expressions, or JSON format checks. Cheaper, faster and free of bias.

What to record for each run

For each question, each model, each run, record one row: question ID, model name and version, reasoning level, the prompt used (or its version ID), the answer, pass or fail, input and output tokens, response time. A spreadsheet is enough to start.

From that table you can work out the numbers that actually drive the decision: pass rate per task type (not just overall), the number of questions unstable across runs, cost per passing question, and response time at the 90th percentile — because customers remember the longest wait, not the average one.

Keeping the evaluation set alive

The most valuable evaluation set is not one built once for a model choice, but one that is kept and rerun:

• Store it like source code: questions, answers, criteria and each run's results, all versioned.
• Add every new failure: whenever the system gets a real case wrong, add it. After a few months, the set contains exactly the hardest situations your business faces.
• Rerun on every change: a new model, an edited prompt, a vendor shipping a new version under the same name. A small prompt change to fix one error can break three other cases — the evaluation set is the only way to see that before customers do.

Conclusion

Public leaderboards are useful for shortlisting three or four candidates worth testing. But the final decision should rest on your own questions, with enough of them that the result is not luck.

Three numbers to remember: 30 questions to rule out clearly weaker models and compare costs; about 100 to separate models a few points apart; run each question three times because the same model does not answer the same way every time. And one principle: write the answers before you look at the responses.

If you want to build an evaluation set from your business's real data — from choosing questions and writing grading criteria to automating reruns whenever a new model appears — contact us or see KonexForge's Data Analytics capability.