The benchmark you are procuring against is a dataset nobody audited

The slide carries one number. A supplier is presenting a system that writes queries against the warehouse, and the evidence offered is a position on a public leaderboard — near the top, a few points clear of the alternatives. Somebody asks how the number was produced. The answer is the name of a benchmark, and the room accepts it, because a benchmark is the kind of thing a room is supposed to accept.

What that number measures is agreement with a set of answers a person wrote down and marked correct. The set is a dataset. It was assembled by annotators working at pace, over databases they did not design, in domains they were not hired for. In most cases nobody has re-read it since.

The claim here is narrow and it is checkable. When somebody does re-read one, the labels turn out to be wrong often enough that the ranking they produced does not survive being corrected. A leaderboard position is not primarily evidence about a supplier. It is evidence about a dataset that the buyer has never seen.

Somebody checked

Four researchers at the University of Illinois audited the annotations in two of the most heavily used text-to-SQL benchmarks and published the result in Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards in January 2026. They report annotation errors in 263 of the 498 examples in BIRD Mini-Dev, and in 76 of the 121 examples in Spider 2.0-Snow — 52.8 per cent and 62.8 per cent respectively. Their code and corrected data are public, which is the part that makes the finding arguable rather than merely alarming.

Error rates alone would be a quality complaint. The consequence is the second half of the work. The authors corrected a sample of one hundred examples from the BIRD development set and re-ran all sixteen open-source agents listed on the benchmark’s leaderboard against both versions, reporting relative performance changes from −7 per cent to 31 per cent and ranking movements of up to nine positions in either direction (Jin and colleagues, 2026).

One result in that paper deserves to be read twice. Rankings measured on the uncorrected sample tracked rankings on the full development set closely, at a Spearman correlation of 0.85. Rankings measured on the corrected sample did not, falling to 0.32 and losing significance (Jin and colleagues, 2026). The uncorrected benchmark was internally consistent. It agreed with itself, reproducibly, across a larger sample of the same flawed annotations.

A measurement that reproduces is not the same as a measurement that is right.

This is not an isolated finding about one dataset. The Agentic Benchmark Checklist, assembled in 2025 by researchers across several institutions, documents the same failure in different clothing across widely cited agent benchmarks — insufficient test cases, empty responses scored as successes — and estimates that such issues distort reported performance by as much as one hundred per cent in relative terms. The pattern is consistent. Benchmarks are built by people who need them to exist, under the same pressure as everyone else, and then they are used as though they were instruments.

Why the errors are invisible from where you are sitting

The useful part of the Illinois work, for a buyer, is the taxonomy rather than the headline rate. The authors sort annotation errors into four patterns, and the patterns differ in what a reviewer needs in order to see them at all.

One class is the one anybody can catch. Where the recorded query contradicts the question in plain terms — an inclusive range where the question asked for a strict inequality, a formula that computes something other than what was requested — the error is visible from the two texts alone. The largest class is not. Errors that come from a limited understanding of the schema or the data account for the majority of flawed examples in both benchmarks: a join on a non-unique key with no deduplication, a missing aggregation, a filter left out because a column was judged unimportant (Jin and colleagues, 2026). Those are only detectable by running queries against the actual database.

1. Level 01 The question and the recorded answer Catches an answer that contradicts what was asked. Needs only reading.
2. Level 02 The schema Catches joins and aggregations that are wrong in structure.
3. Level 03 The data Catches answers wrong because of what is in the tables. The largest class.
4. Level 04 The domain Catches a label that encodes an incorrect fact about the subject.
5. Level 05 A ruling on ambiguity Catches a question with more than one defensible answer, and needs an owner to settle it.

Read that ladder as a statement about access. An evaluation set published as a score, without the database behind it, can be audited at the first level and nowhere below. The majority of its errors sit below. This is the mechanism by which a benchmark number remains credible for years: not because it was verified, but because verifying it requires something most of its users do not hold.

What to do with a benchmark number in a procurement

None of this argues for discarding public benchmarks. It argues for demoting them from evidence to context, and for moving the evidential weight onto a set you can inspect — which, in an enterprise, means your own schema, your own questions and your own answer key.

The last of those is the one teams skip and the one that costs most later. Nearly a third of the flawed BIRD examples were flawed because the question itself permitted more than one reasonable reading (Jin and colleagues, 2026). An answer key built over your own data will inherit exactly that problem, and the resolution is not technical. Somebody has to decide what the business means by an active customer, and that decision has to be written down next to the answer, because it is part of the answer.

There is a regulatory edge to this as well. Article 15 of the EU AI Act requires that the levels of accuracy and the relevant accuracy metrics of a high-risk system be declared in the instructions for use. A number that began life as a leaderboard position can end up inside a declaration, at which point the provenance of the set it was measured on stops being an engineering preference. Whether any particular system falls in scope, and what a declaration has to say, is a question for counsel — this practice works on readiness and on the evidence that supports it, and does not certify anyone.

What this does not tell you

The audit covers two benchmarks in one task family. Text-to-SQL is unusually exposed, because the answer key is executable and therefore checkable at all — which is the reason these error rates are known rather than the reason they are high. For a summarisation or classification benchmark there is often no equivalent way to run the check, and the honest position is that the error rate is unmeasured rather than low.

Nor does any of this establish that the suppliers at the top of a leaderboard are the wrong choice. Correction moved rankings substantially in the study, and it moved some agents up. The finding is about the informativeness of the ordering, not about the direction of any particular error. A system may well be the right one and the number may still not be the reason.

And this is not the argument that the number should be your own instead of theirs. It is the argument that you need one you can open. We have made the adjacent case before, that a statistical baseline is worth establishing before the expensive model arrives — this is the same discipline pointed at the answer key rather than at the model.

The person this changes is whoever writes the evaluation section of the next supplier assessment. The current draft almost certainly names a benchmark and a threshold to clear on it. The version worth signing names an evaluation set that your organisation owns, a count of how many of its answers have been checked by a second pair of eyes, and the name of the person who rules when a question turns out to have two answers. That document takes a fortnight of unglamorous work to produce, and it is the only part of the assessment that will still mean something when the leaderboard is rebuilt.