Journal — The Right-Sized LLM Benchmark

Good enough is defined by the judged score on the specific measured task, in the execution mode—batch or sync—that the workflow actually uses, plus whether the model answers at all. A high score that requires salvage, retries, or manual repair is a poor production choice. This is one axis of compari

Since the previous cycle’s routing guidance for Qwen3.7 and Nemotron tiering, the current measurement introduces Claude Sonnet 5, minimax/minimax-m3, and Grok-4.5 across topic discovery, SEC filing interpretation, author-voice reproduction, and content drafting, with no reported score reversal or pr

There is no defensible universal winner across relevance scoring, classification, prioritization, and matching. The supplied results do not state how many tasks are in this category or how many models are measured on them, so those counts cannot be reported without inventing evidence. They do show a

For Infrastructure & Utility Work—work that names topic clusters, determines topic sequence, scores report relevance, analyzes filing chunks and full filings, writes analyst content tied to cited claims, refines claims, summarizes content, generates executive summaries, performs synthesis analysis,

For Financial Analysis & Trading Decisions, the practical quality bar is 8.0 out of 10. The evidence shows a trade-off rather than a universal winner: Grok-4.5 offers the strongest measured balance of trading-recommendation quality and synchronous cost, while GPT-5.5 and Gemini-3.5-flash are credibl

For this category, “good enough” is not a single headline score. A production model must deliver sufficiently strong output for the intended workflow—whether turning source material into analysis, summarizing content, generating sections, or filing chunks—while also meeting required standards for cl

The current evidence does not support a universal model ranking. It supports routing decisions by workflow stage, quality bar, serving mode, cost, and the probability that a request produces a usable answer. A conditional quality score is not enough when the model frequently fails to answer, returns

Production routing is a portfolio problem. The current evidence covers 54,627 scored judgments across 910 measured model-and-task combinations, covering 26 models on 52 tasks; the figures were published on a single date, and the measurement behind them is continuous. No single model wins enough of t

A model is good enough for production only when it clears the required quality bar and produces an accepted result reliably enough for the workflow. The measurements here show why those conditions must be evaluated together: a high score describes the quality of returned work, but not necessarily th

For production use, “good enough” is not simply a high score on answers that a model manages to return. The relevant bar is usable output on the first provider call: the response must arrive, conform to the workflow’s expected structure, and avoid salvage or a second repair call. The current evidenc

This snapshot’s broadest measured base for a single model belongs to Kimi-k3—not because it is the universal leader, but because the evidence covers it across demanding organization and interpretation stages with explicit score intervals, costs, and conditional measurements. I choose it as the deep-

For engineers selecting production models for topic organization and clustering, a practical quality bar is **8.0 out of 10**. The current excerpts cover naming clusters, open-ended discovery and theme generation, topic sequencing, section assignment, document-relevance scoring, client-topic matchin

The supplied results do not specify a universal quality threshold, nor do they state how many tasks or models make up the full Structured Data & Fact Extraction category. This comparison therefore uses a practical, task-level bar: a model must meet the extraction quality required by the workflow, wh

The supplied evidence does not state how many tasks are in the Social & Promotional Content category or how many models are measured across them. It also does not define a single category-wide quality threshold. The defensible production bar is therefore task-specific: choose the highest-quality opt

There is no defensible single winner across the long-form content measurements supplied here. The category-level task and model counts are not stated in the available evidence, so this comparison cannot provide them without inventing a figure. The results instead support a practical quality bar of *

Claude Sonnet 5 clears the 90% quality bar on 22 of 52 benchmark tasks, with real strength in financial analysis, extraction, and long-form writing. But it wins zero cheapest-good-enough slots, so the bench frames it as a premium quality pick, not a default value router.

Blended cost is the harness's final number — token cost, token shape, lane, and success rate folded into one honest price per useful result. This finale opens the box, shows the whole machine inside it, and closes the loop: the bench is the harness's readout, and its proof.

The list price of an LLM call is marketing. The invoice is truth. This is the harness's cost-measurement component — how it closes the loop between the rate card and the bill, and why the ★ "cheapest qualifier" means something because of it.

Multiplying tokens by the headline rate feels like the obvious way to compare model costs. It's wrong in three structural ways — and the harness's cost component exists to catch every one, so the price ranking the bench reads out is honest instead of the rate card's fiction.

Inside a mature harness, a model is an interchangeable part — and new parts arrive faster than organic traffic could ever score them. This is how the production harness places a brand-new model on the 0–10 scale quickly, by replaying real past work through it, and how deterministic sampling keeps the steady-state judging affordable.

Two models can post the same average score and still not be equally trustworthy. The confidence band is how the harness tells them apart — how it names what it doesn't yet know, gates which models earn live traffic, and why some cells on the bench show nothing at all.

The harness's evaluation layer is only as trustworthy as its judges — which are LLMs too. Here's the machinery that makes it honest: multi-judge consensus, cross-provider exclusion, bias correction, drift detection, and rehearsed panel changes — turning "trust us" into "audit us," with the panel published dated on the bench.

"Quality" means two different things depending on the task, so the harness's evaluation component measures it two ways and combines them: deterministic verifiers catch what code can see, a judge panel grades what it can't, and a task-specific rubric turns the result into a single 0-10 score relative to the job the harness was built to do.

Two parts of the harness that keep it honest: latency-tolerant work runs at a steep discount in a batch lane, so the bench quotes both prices for the same interchangeable model — and every result is pinned to the exact prompt version that produced it, so scores never silently drift.

Every success rate on the bench is the readout of the harness's reliability layer — the machine around the model that catches malformed responses, classifies non-model failures, reroutes around broken providers, and refuses to count a model's misfires against it unfairly. Here's how that layer works.

Routing is the harness's model-selection component: every ★ on the heatmap is one runtime decision — pick the cheapest model that has earned the right to do this job. It works precisely because the harness holds prompts, validation, evaluation and cost fixed and swaps only the model, an interchangeable part. Here's the two-gate decision, and what catches it when it's wrong.

Leaderboards rank models. A harness is built around a job — its task definition — and ranks model-on-that-job. This is the first component of the machine, the fixed thing everything else attaches to, and why the same model can be your cheapest pick for one task and not even qualify for the next.

In Part 1 we argued the enterprise question isn't "which model is best?" but "which is good enough for the task?" This is the harness that answers it — the complete task-specific machine around the model, of which the bench is the visible readout — and the map for everything we dig into next.

Most enterprise AI teams ask which model is best. In production the better question is which model is good enough for a specific task — and the answer isn't a model at all, it's a harness: task-specific infrastructure that decomposes the work, holds the quality bar, and treats models as interchangeable hires. This origin post argues for task-level benchmarking as the honest readout of that harness.