How Do You Put a Number on Quality? (5 of 11)

Ask a model to return a customer record as JSON and you can check the answer with a script. Ask it to summarize a quarterly filing for a busy executive and no script on earth can tell you whether the summary is faithful. Both are “quality.” They are not the same kind of quality, and pretending they are is how benchmarks lull buyers into trusting numbers that don’t survive contact with production.

This is the problem the evaluation component of the harness exists to solve. A harness is the complete task-specific infrastructure wrapped around a language model to make it reliably do a defined job; evaluation is the part of that machine that decides whether a given output was good enough. In the harness behind “good enough” we sketched the whole apparatus, which produces a single quality score per task and model. Here we open the hood on how that score gets made — and why it takes two mechanisms, not one, to make a quality number you’d stake a decision on.

See it live: the 0–10 quality score on every task page comes from what this post describes — see how it’s built, verifiers and judges together.

Two kinds of “good,” two ways to measure

Some things about an answer are objective. The JSON either parses or it doesn’t. The required fields are either all present or one is missing. The numbers in a table either add up or they don’t. A citation either survived a rewrite intact or it got mangled. In a medical transition-of-care handoff, the dosage is either exactly right or it is a patient-safety incident.

Other things are irreducibly qualitative. Is the summary faithful to the source, or did it quietly invent a number? Is the tone appropriate for the audience — plain for a layperson, precise for a specialist? Does the reasoning hold together, or does it sound confident while being wrong? In that same handoff, does the language carry the empathy a frightened family needs to hear?

No single tool measures both. So the harness uses two, deliberately. Deterministic where you can be, judged where you must be. That hybrid is not a compromise — it’s the whole point.

The verifiers: code that doesn’t have opinions

The first half is a set of deterministic checks. These are programs, not prompts. They return pass or fail with no ambiguity and no mood. Did the output validate against the required schema? Are the figures internally consistent? Did every citation survive the rewrite? Is the dosage correct to the milligram?

The virtue of code here is that it is cheap, unambiguous, and tireless. A human reviewer skimming a hundred outputs will miss the one summary that flipped a “increase” to a “decrease.” A verifier checking internal consistency will not. These are the silent failures that matter most precisely because they look fine — the output is fluent, well-formatted, plausible, and quietly wrong. Verifiers are built to catch exactly that.

There’s a second job these checks do for free. A verifier failure is a clean, trustworthy signal. When an answer fails an objective check, the harness doesn’t just dock the quality score — it has grounds to escalate, to try a different model on the same work. That’s the seam between the evaluation component and the reliability components around it: the same check that grades an output also tells the runtime when to reach for a stronger one. We go deeper on that in engineered reliability and on the live decision in picking the model at runtime.

But verifiers are blind to everything that isn’t mechanical. A perfectly valid JSON blob can summarize a document dishonestly. Code can confirm the shape of an answer is correct. It cannot confirm the answer is good. For that you need judgment.

The judge panel: grading what code can’t see

The second half is a panel of judges. The harness takes the qualitative dimensions — faithfulness, tone, soundness of reasoning, empathy where the task calls for it — and has several strong models grade the output independently. Then it takes the consensus.

The reason it’s a panel and not a single judge is the obvious one: any one model is noisy, has blind spots, and can be gamed. One grader’s quirk shouldn’t decide a model’s fate. A handful of capable models, scoring independently and agreeing, is far harder to fool than any of them alone. The consensus is the signal; the disagreement is information too.

Making those judges actually trustworthy — keeping them independent, never letting a model grade its own work, correcting for their individual biases — is a discipline in its own right, and it’s the subject of the very next post. For now, hold this thought: a judge panel is only as good as the rules it follows, and a great deal of care goes into those rules.

The number is relative to the job, not to “smartness”

Here is the part that separates this from a leaderboard. The verifiers and the panel combine into a single 0-10 quality score for each task-and-model pairing. And that score is relative to the task’s own definition of good — not some generic measure of how clever the model is. This is what makes the evaluation component task-specific rather than model-specific: it grades an output against the job the harness was built to do, so the model inside is just an interchangeable part being measured against a fixed standard.

Every capability can carry a tailored rubric: an explicit statement of what matters for this work and what to ignore. The instruction to “ignore cosmetic JSON formatting” matters when that formatting gets normalized downstream anyway — a judge that dings a model for whitespace is measuring noise. Or consider a task that, by design, reschedules the items it can’t get to in one pass; for that task the rubric tells judges to grade only the items actually delivered and to treat completeness as out of scope. Penalizing the model for work it correctly deferred would be measuring the wrong thing entirely.

This is why a generic benchmark and a work-package benchmark diverge so sharply. A model that aces math olympiad problems may be mediocre at writing an empathetic handoff; a model that writes beautifully may quietly fail a strict dosage check. The rubric is what makes the resulting number mean something for the job you’re actually buying the model to do — which is exactly the argument we made in why we benchmark work packages. And because the rubric shapes the score, we publish it. You can read what each task was graded on.

Why neither half works alone

It’s worth saying plainly why the harness doesn’t just pick one mechanism and simplify.

An evaluation that is all judges is soft. Judges are persuadable. A fluent, confident, well-structured answer can charm a panel into a high score even when a hard check would have caught it cheating. Lean only on judgment and you’ve built something manipulable.

An evaluation that is all code is blind. Verifiers can’t read faithfulness, tone, or reasoning. Lean only on deterministic checks and you’ll happily certify a model that returns immaculate JSON full of plausible fabrications.

The hybrid closes both gaps. Code nails down the objective floor — nothing malformed, nothing inconsistent, nothing that fails a hard safety check gets a pass. The panel measures the qualitative ceiling — how faithful, how sound, how fit for the audience. Together they produce a number that is hard to game and sensitive to real quality. That robustness is the entire reason to build the evaluation component the harder way.

Where the score lives

The bench is the visible readout of the harness — a dated snapshot of what the evaluation component produced when prompts, validation, and cost were held constant and only the model varied. There, that 0-10 score and the rubric behind it are visible for every task. The quality bar — the slider you set as a buyer — sits on top of the score, expressed as “at least X% of the best model’s score for this task.” Raise the bar and you demand answers closer to the frontier; relax it and cheaper models qualify. The score is the substrate; the slider is your knob.

But a score alone, however carefully made, isn’t the end of the story. Two questions sit right behind it. First: who do we trust to produce these scores — what keeps the judges honest? And second: how sure are we a given score is real and not an artifact of too few samples? A 7.8 from two grades and a 7.8 from two hundred are not the same claim.

We answer the first next, in who watches the judges — the part where the panel earns its trust. The second comes after, in confidence, not just scores, where a number learns to admit what it doesn’t yet know.

If you’ve ever shipped an LLM feature that “tested fine” and failed quietly in production, the gap was almost certainly something a verifier would have caught. Next: how the harness keeps the judges themselves honest.