An automated judge is an instrument. Calibrate it or do not read it.
Scoring model outputs with another model has become the default evaluation method, and the published evidence says these judges can be highly repeatable while being systematically wrong in ways repeatability will never reveal.
The evaluation harness is the part of the system that everybody trusts and nobody audits. It produces a number, the number goes up when the team improves something, and it is the number that appears in the slide justifying the release. Somewhere underneath it, a second model is reading each output and deciding whether it is good.
That arrangement is reasonable. Human evaluation does not scale, and an automated judge is the only way most teams can evaluate anything at all at the frequency release cycles now demand. What is not reasonable is treating the judge as a measuring device that has been calibrated, when in almost every case it has not been.
The claim: an automated judge is an instrument with known systematic errors, and the errors do not show up in the consistency checks teams actually run.
Consistency is not accuracy
The natural way to check a judge is to run it twice and see whether it agrees with itself. It usually does, and that is the trap. A large study across twenty-one judge models and several hundred thousand individual judgements reported exactly this pattern — very high test-retest reliability coexisting with severe positional bias in judges already deployed in production. The paper, Reliability without Validity, is worth reading in full if your release gate depends on one of these.
Positional bias is the best-studied of the family: presented with two candidate answers, judges tend to prefer one position over the other regardless of content. It has its own literature — the systematic study of position bias in LLM-as-a-judge sets out how it is measured — and it is joined by preferences for longer answers, for answers written in a familiar style, and for answers produced by the judge’s own model family.
Every one of those biases is stable. A stable bias is invisible to a repeatability check and fatal to a comparison, which is precisely what an evaluation harness is used for.
What the errors do to a decision
| What the harness reports | What may actually be true |
|---|---|
| Variant B scored higher than variant A | Variant B was shown second, and this judge prefers the second answer |
| Quality improved after the prompt change | The change made answers longer, and this judge rewards length |
| Our own model outperformed the alternative | The judge shares a family with our model and prefers its phrasing |
| The score held steady across the release | The failure mode that appeared is one the rubric never asked about |
The last row is the one that costs most and gets discussed least. A judge scores what the rubric names. Anything the rubric does not name — a tone that is wrong for the audience, a citation to a document the user cannot access, a confident answer to a question that should have been refused — is not scored as bad. It is not scored at all, and the aggregate is unchanged.
What calibration actually involves
It is less work than it sounds, and it is a fixed cost rather than a recurring one.
The human-labelled set is the part teams resist and the only part that makes the rest meaningful. Without it there is no ground truth, and every subsequent comparison is between two model opinions with no reference point. A few hundred carefully labelled items — assembled once, versioned, extended when a new failure mode appears — is enough to state an agreement rate and to notice when it moves.
Position swapping is nearly free and removes the single largest known bias. If the two orderings disagree on a pair, that pair is a tie, and a harness that records ties honestly reports a smaller difference than one that does not. That smaller difference is the true one.
The last line is the one that catches teams out in production. The judge is a hosted model. It changes. When it changes, every historical score becomes incomparable with every new one, and unless the version is pinned and the change is noticed, the first sign is a step in a metric that nobody can explain.
What this does not tell you
This is not an argument against automated judging. Run at volume, with the controls above, it is the most useful evaluation tool available to a team shipping regularly, and the alternative for most organisations is not human evaluation but no evaluation.
It is also not a claim that any particular published agreement figure transfers to your task. It does not. Agreement is a property of the rubric, the domain and the population of outputs, and a judge that tracks human preference well on general assistant responses may be close to useless on clinical summaries or contract clauses.
The reader who acts on this is whoever presents the evaluation number at a release gate. Before the next release, be able to answer one question: how well does the judge agree with a person, on our data, and when was that last measured. If the answer is that nobody has checked, the release gate is a formality with a chart attached.