Who Watches the Judges? (6 of 11)

Here’s the objection that should occur to anyone reading a benchmark like ours, and that we’d rather raise ourselves than have you raise for us: you use LLMs to grade LLMs. So who judges the judges?

It’s a fair hit. A score is only as trustworthy as the thing that produced it, and if the grader is the same kind of system as the contestant — prone to the same blind spots, the same stylistic preferences, the same overconfidence — then the whole leaderboard is just one model’s taste dressed up as measurement. A benchmark that waves this away with “our judge is very capable” hasn’t answered the question. It’s hoped you won’t ask it.

This is the evaluation layer of the harness — the ninth component of the machine we’ve been describing across this series, the complete task-specific infrastructure that surrounds a model to make it reliably do a defined job. Evaluation is the part that decides what “reliably” even means: it’s how the harness knows which model to trust for the work, and how it keeps knowing as models change underneath it. And that layer is only as good as its judges. So this post answers the objection, mechanism by mechanism. The short version: the reliability of the evaluator is itself something you engineer and measure — the same philosophy the harness applies to the models on the heatmap. The longer version is five guardrails and one habit. The habit is publication: we put the judge panel on the bench, dated, so you don’t have to take any of this on faith. (For where these scores come from in the first place, see Measuring quality; this is the companion piece — not how the harness grades, but why you can trust who grades.)

See it live: the judge panel is published in full on the bench — open it and check who’s doing the grading before you trust the grades.

No single judge decides anything

The first and biggest fix is structural: there is no “the judge.” Quality on a given output isn’t a verdict handed down by one model. It’s the aggregate of a panel — several strong models from different labs, each grading the same output independently, against the same published rubric, without seeing each other’s scores.

Then the harness combines them — a central tendency across the panel, a mean or median rather than any one vote. The point is variance reduction. Every model has quirks: one over-rewards fluent prose, one is a stickler about format, one is a soft touch on factual slips. Average across a diverse panel and the idiosyncratic noise starts to cancel, while the real signal — is this output actually good? — survives. No single judge’s bad day, or pet preference, can swing a verdict on its own.

This is also why a wider panel doesn’t just produce a different number — it produces a steadier one. More independent reads, less sampling jitter. That steadiness matters downstream, and we’ll come back to it.

A model never grades its own lab’s homework

The most glaring conflict of interest in LLM-as-judge is the obvious one: a lab’s model grading that lab’s own output. Even with no ill intent, models from the same family share training lineage, share stylistic instincts, share failure modes. A judge that “recognizes” its sibling’s style as good is not a fair referee — it’s a character witness.

So the harness forbids it outright. For any candidate model being scored, every judge from the same provider is removed from that candidate’s panel. If it’s grading a model from Lab A, no Lab-A model sits on the panel for that grading. Each contestant is judged only by outsiders.

It’s a blunt rule, and deliberately so. You could imagine subtler corrections, but bluntness is a feature here: it’s easy to state, easy to verify from the published panel, and it closes the single most embarrassing loophole — a model grading its own homework — without relying on anyone’s good faith.

Lenient and harsh judges, corrected before they count

Even honest, independent judges aren’t neutral instruments. Run enough gradings and the personalities emerge: one judge is consistently lenient, handing out eights where the panel says six; another is consistently harsh, docking points everyone else waves through. If you just average raw scores, the lenient judge quietly inflates every verdict it touches and the harsh one drags things down — and which models happen to draw which judges starts to matter more than it should.

So the harness measures each judge’s systematic tendency and corrects for it before aggregating. Think of it as a per-judge zero-point adjustment: if a judge runs half a point hot on a given kind of task, that bias is subtracted back out before its score joins the panel average.

Two details matter. First, the correction is per judge, per task — not one global knob. A model can be lenient on long-form writing and dead-on for structured extraction; a single blanket adjustment would break one to fix the other. Second, it’s an offset, not a thumb on the scale: what’s removed is a known, measured tilt, so the judge’s real discrimination — its ability to tell good from bad — comes through cleanly. The harness is calibrating the instrument, not deciding the answer in advance.

Judges are watched for drift

Calibration handles the biases you can see today. But a judge panel is a living thing — model endpoints get updated, providers change defaults, and a grader that behaved one way last month can quietly start behaving differently. An uncalibrated drift is more dangerous than a static bias, because a static bias gets corrected and a drift just accumulates.

So the judges are supervised, continuously, the way the harness supervises any production component it depends on. It tracks signals like: how often each judge agrees with its peers, how often it refuses or returns something unusable, and whether either of those is trending over time. A judge that suddenly starts disagreeing with the rest of the panel, or refusing a class of inputs it used to handle, or sliding steadily in one direction, gets flagged for review.

The principle is the one the harness applies to everything on the bench: a number you can’t see going wrong is a number you can’t trust. Watching the watchers isn’t a one-time audit — it’s a standing measurement, with the judges held to the same “prove it, keep proving it” bar as the models they grade.

Changing a judge is the riskiest edit we make — so we rehearse it

Here’s a subtle danger that’s easy to miss. Swapping a judge model — retiring one, promoting a new one onto the panel — is one of the most destabilizing edits the harness can make. A new judge with even a slightly different temperament could shift every score on the board at once, silently, and there’d be no clean way to tell a real ranking change from an artifact of the new referee.

So the harness doesn’t just swap a judge and hope. Before a new judge goes live, it replays past gradings through it on a shadow panel — re-running historical work it never saw, and comparing its verdicts against what the current panel concluded. If the candidate judge tracks the existing panel acceptably and shows no alarming new bias, it gets promoted. If it diverges, it doesn’t go live until we understand why. Changing the measuring stick is treated as a controlled experiment, not a config tweak — the same interchangeable-part discipline the harness applies to the contestants, turned on the referees.

And — the habit again — when the panel does change, the bench records it: the new composition, the date, and the reason. A score from March was produced by March’s panel; a score from June by June’s. The measuring instrument is versioned right alongside the measurements.

We publish the panel — that’s the whole point

Every guardrail above is, in the end, in service of one thing you can do yourself: look.

The bench publishes a dated snapshot of the judge panel — which models are grading, in what roles, as of that snapshot. Not “a panel of frontier models.” The actual composition, frozen with the rest of the snapshot, next to the rubrics and the example inputs and outputs. If you think a particular grader shouldn’t be on the panel for a particular task, you can see that it’s there and argue the point. If you want to know whether a lab is grading its own work, you can check. The answer to “who watches the judges?” is, ultimately, you can — because we showed you who they are.

That’s the difference between “trust us” and “audit us,” and it’s the only honest stance to take here. A lab harness — a fixed set of curated prompts, run once, scored, and ranked — never has to answer this question, because nobody depends on its judges past the moment the ranking is published. A production harness runs messy, single-pass work all day, and its evaluation layer is load-bearing: get the judges wrong and every routing decision downstream inherits the error. So the bench is the readout that proves the harness’s judges are honest — a dated snapshot of the evaluation layer, held to the same standard as the work it governs. Let’s be candid: LLM-as-judge is an active, imperfect, fast-moving area of practice. We are not claiming the judges are flawless. We’re claiming something narrower and more defensible — that the harness treats the evaluator as a system to be engineered and measured rather than assumed: independent panels, conflicts excluded, biases corrected, drift watched, changes rehearsed, and the whole arrangement published so you can check our work. A benchmark is only as trustworthy as its judges. So we built the judges to be inspected.

There’s a thread running out of this post worth pulling. Notice that nearly every guardrail here — the multi-judge panel, the bias offsets — does the same quiet thing: it reduces variance. Less noise in the grading means a tighter interval around each model’s score, which is exactly what lets a verdict graduate from “we think” to “we’re sure.” That machinery of confidence — why every score on the bench travels with an interval, and why the harness hides the cells it’s not yet sure of — is the next piece. And it’s the same calibrated panel, run at volume, that places a brand-new contestant on the scale for the first time in Onboarding a new model.

Next in the series: Confidence, not just scores — why a number without an interval is a guess wearing a lab coat. Open the heatmap and check who’s grading before you trust the grades.