Journal — The Right-Sized LLM Benchmark

Claude Sonnet 5 clears the 90% quality bar on 22 of 52 benchmark tasks, with real strength in financial analysis, extraction, and long-form writing. But it wins zero cheapest-good-enough slots, so the bench frames it as a premium quality pick, not a default value router.

Blended cost is the harness's final number — token cost, token shape, lane, and success rate folded into one honest price per useful result. This finale opens the box, shows the whole machine inside it, and closes the loop: the bench is the harness's readout, and its proof.

The list price of an LLM call is marketing. The invoice is truth. This is the harness's cost-measurement component — how it closes the loop between the rate card and the bill, and why the ★ "cheapest qualifier" means something because of it.

Multiplying tokens by the headline rate feels like the obvious way to compare model costs. It's wrong in three structural ways — and the harness's cost component exists to catch every one, so the price ranking the bench reads out is honest instead of the rate card's fiction.

Inside a mature harness, a model is an interchangeable part — and new parts arrive faster than organic traffic could ever score them. This is how the production harness places a brand-new model on the 0–10 scale quickly, by replaying real past work through it, and how deterministic sampling keeps the steady-state judging affordable.

Two models can post the same average score and still not be equally trustworthy. The confidence band is how the harness tells them apart — how it names what it doesn't yet know, gates which models earn live traffic, and why some cells on the bench show nothing at all.

The harness's evaluation layer is only as trustworthy as its judges — which are LLMs too. Here's the machinery that makes it honest: multi-judge consensus, cross-provider exclusion, bias correction, drift detection, and rehearsed panel changes — turning "trust us" into "audit us," with the panel published dated on the bench.

"Quality" means two different things depending on the task, so the harness's evaluation component measures it two ways and combines them: deterministic verifiers catch what code can see, a judge panel grades what it can't, and a task-specific rubric turns the result into a single 0-10 score relative to the job the harness was built to do.

Two parts of the harness that keep it honest: latency-tolerant work runs at a steep discount in a batch lane, so the bench quotes both prices for the same interchangeable model — and every result is pinned to the exact prompt version that produced it, so scores never silently drift.

Every success rate on the bench is the readout of the harness's reliability layer — the machine around the model that catches malformed responses, classifies non-model failures, reroutes around broken providers, and refuses to count a model's misfires against it unfairly. Here's how that layer works.

Routing is the harness's model-selection component: every ★ on the heatmap is one runtime decision — pick the cheapest model that has earned the right to do this job. It works precisely because the harness holds prompts, validation, evaluation and cost fixed and swaps only the model, an interchangeable part. Here's the two-gate decision, and what catches it when it's wrong.

Leaderboards rank models. A harness is built around a job — its task definition — and ranks model-on-that-job. This is the first component of the machine, the fixed thing everything else attaches to, and why the same model can be your cheapest pick for one task and not even qualify for the next.

In Part 1 we argued the enterprise question isn't "which model is best?" but "which is good enough for the task?" This is the harness that answers it — the complete task-specific machine around the model, of which the bench is the visible readout — and the map for everything we dig into next.

Most enterprise AI teams ask which model is best. In production the better question is which model is good enough for a specific task — and the answer isn't a model at all, it's a harness: task-specific infrastructure that decomposes the work, holds the quality bar, and treats models as interchangeable hires. This origin post argues for task-level benchmarking as the honest readout of that harness.