The Right-Sized LLM Benchmark Cheapest LLM that's good enough.

The Unit of Measurement Is the Job, Not the Model (1 of 11)

A model scores 91% on a famous reasoning benchmark. You wire it into your contract pipeline to pull payment terms into structured JSON. It works beautifully on the three contracts you tested. Then it hits a vendor agreement with the terms buried in a footnote and an addendum, and it confidently returns the wrong number — in valid JSON, so nothing downstream flags it.

Nothing about the 91% predicted that. The benchmark measured a general aptitude. Your pipeline needed this job done on your documents. Those are not the same question, and the gap between them is where a lot of enterprise AI quietly goes wrong.

Part 2 put the fix in a single line: the unit of work is a capability, not a model. That sentence is the foundation everything else in this series stands on. It is also the first component of a harness — the complete task-specific infrastructure that surrounds a language model to make it reliably do a defined piece of work. Every harness is built around one thing before anything else: the job it exists to do. That job, written down and made measurable, is the task definition. This post is that first component, dug all the way down.

See it live: open the bench — every column is a category, every ★ a “cheapest good-enough” pick. This post is why the unit is the job, not the model.

What a leaderboard actually measures

A leaderboard gives you one number per model. It’s an aptitude test — a standardized exam the model sits once, scored against a fixed answer key.

Part 1 made the analogy that sticks: nobody hires a customer-service rep off their SAT score. The SAT correlates with something, but you wouldn’t stake a hiring decision on it, because the test isn’t the job. It’s clean, it’s curated, and it rewards the kind of careful, multi-step problem-solving a lab can stage. Your inbound-message classifier doesn’t get a quiet exam room and three attempts. It gets a messy message, one pass, and a structured label your routing logic depends on.

And the headline exams are mostly saturated anyway. The big multiple-choice suites sit north of 88%; the math sets are near 99%. When the top of the field is bunched within a point or two, the ranking stops discriminating exactly where you need discrimination most. A leaderboard tells you a model is broadly capable. It does not tell you whether it does your work, reliably, at a price you can run at scale.

A capability is the job the harness is built around

So we measure something narrower and far more useful: a capability. A capability is the harness’s task definition — the fixed thing the whole machine is assembled around, the job the model is being asked to perform.

A capability is one defined work package. It has three parts, and all three are fixed:

  • a fixed input shape — what comes in (a contract, an article, a long report, an inbound message);
  • a fixed required output shape — what a downstream system actually consumes (a JSON object with named fields, a number between 0 and 1, exactly five bullets, one category label);
  • a real quality bar — how good the output has to be before it’s safe to use.

Fix those three and you’ve drawn the outline the rest of the harness fills in — the prompts, the validation, the routing, the evaluation all attach to this input, this output, this bar. The model, notably, is not in that list. It’s the one part the harness leaves interchangeable: swap model A for model B and the input shape, the output shape, and the bar don’t move an inch. That’s the point of defining the task first — everything that matters about the job is nailed down before a single model is chosen.

Concretely, these are jobs like: extract the payment terms from a contract into structured JSON your systems can read; score an article’s relevance to a topic from 0 to 1; compress a long document into five bullets a busy person can trust; classify an inbound message; write a platform-native promo post. On the bench, each of these is a task, and a task page shows you which models clear that task’s bar and what each one costs to run.

The difference from a leaderboard is what we call ecological validity — the test resembles the real world closely enough that passing it predicts real performance. This is also the difference between two kinds of harness. A lab harness is a fixed set of curated prompts, run once, scored, ranked — clean, staged, useful for mapping the frontier, and lived-in by every headline benchmark. A production harness is the machine that runs messy, single-pass work all day. Three things make that gap concrete:

  • Real, messy inputs. Not curated exam questions — the actual domain-specific documents and messages that production sees, footnotes and addenda included.
  • Single-pass, zero-shot. No multi-step scaffolds, no “let the model try five times and keep the best.” One shot, the way a high-volume pipeline runs it, because that’s the cost and reliability profile you actually pay for.
  • Outputs a system consumes. The answer has to be structurally usable by the next step. “Mostly right prose” doesn’t pass when the next stage needs valid JSON with every field present.

That last point is why a 91% global score and a stumble on your contract format live together so comfortably. The lab-harness exam never required the model to fill your schema from your documents. The production harness’s task definition does.

Every model-on-a-job has its own track record

Here’s the move that makes this more than a renaming exercise. The atom we measure is never “a model.” It’s a pairing: one capability combined with one model. Because the capability is fixed and the model is the interchangeable part, each pairing is really “this task definition, running on this particular model” — and each keeps its own independent record: its own success rate, its own quality history, its own characteristic cost.

That independence is the whole point. A model isn’t good. A model is good at a capability, or it isn’t. Two different jobs can — and routinely do — rank the same set of models in completely different orders.

A model can be precise and literal in a way that makes it superb at field-level extraction and merely adequate at sustained, voice-consistent long-form writing. Another can be a gifted writer that’s careless with exact figures. Their leaderboard ranks might be a point apart. On the jobs you care about, they’re not interchangeable at all — and only a per-pairing record can show you that. Collapse it into one number per model and you’ve thrown away the exact signal that makes the decision.

Eight categories, eight different bars — ours, not a standard

Tasks don’t float in isolation. We group them into categories, and the grouping carries real meaning, because different kinds of work hold different quality bars and react differently to price. The eight below are ours — the natural seams in the particular workload our own harness runs, drawn from our production data, not a taxonomy anyone has to inherit.

  • Financial analysis & trading decisions — read business and financial documents deeply enough to make forward-looking judgments.
  • Structured data & fact extraction — precise, field-level retrieval into a fixed shape.
  • Content summarization & synthesis — compress long input down to the essential without distorting it.
  • Long-form content generation — sustained composition that holds a voice across length.
  • Social & promotional content — concise, platform-native, engaging.
  • Relevance, classification & matching — semantic similarity and category judgment.
  • Topic organization & clustering — discover the natural groupings in a pile of items with no schema handed to you.
  • Infrastructure & utility — mechanical format conversion and metadata work.

And this is the part to hold onto: these eight are an example, not a standard you have to adopt. They’re simply the shape our own harness’s workload happens to take. Another organization’s pipeline would carve into entirely different categories — contract review, ticket triage, code review, claims adjudication, whatever its real work happens to be. The specific list doesn’t transfer, and it isn’t meant to. What transfers is the principle underneath it: every distinct kind of work sets its own quality bar and its own cost sensitivity, and you can only route intelligently once you’ve named those kinds — once each is written down as a task definition of its own. We publish ours so the idea is concrete — not so you’ll copy the labels.

This is the portfolio of tasks idea from Part 1, made into structure. A forward-looking financial judgment carries an unforgiving accuracy bar and tolerates a higher price, because a wrong call is expensive. A mechanical metadata conversion runs millions of times and is brutally cost-sensitive; a small per-call premium becomes a budget. So a model can be the cheapest qualifier — the one wearing the ★ on the heatmap — in one category, and not even qualify in another. The heatmap is models down one axis and our categories across the other, precisely so you can read that pattern at a glance instead of trusting a single rank — and the same picture would redraw itself around whatever task definitions your own work splits into.

When one job wears several outfits

A small wrinkle worth naming, because it’s easy to misread. Sometimes a single capability is exercised through several surface variants. A short-promo capability, for instance, might serve a handful of social platforms, each with its own length limit and conventions.

We treat these as one capability — one task definition — when they exercise the same underlying skill: the same kind of reasoning, the same shape of output, the same failure modes. The reason is purely about signal: split essentially-identical work into separate scorecards and you fragment the evidence, so each fragment takes far longer to reach confidence. Pool work that’s genuinely the same skill and the evaluation signal stays strong, and a model earns a trustworthy verdict sooner. The test of whether to pool is never “same prompt” — it’s “same job, in the way that matters.”

Why this is the atom

Define the unit right and the rest of the harness has something solid to stand on. Define it wrong and every number above it inherits the error. Task definition is the first component for a reason: nothing else in the machine — no prompt, no validator, no routing rule — has a fixed reference point until the job is nailed down.

Measure the model, and the best you can ever produce is an aptitude score — interesting, directionally true, and not something you’d stake a production decision on. Measure the job — a fixed input, a required output, a real bar, tracked per model with its own history and its own cost — and what you get out the other side isn’t a ranking. It’s a hiring decision: for this work, this model is good enough, and it’s the cheapest one that is. And because the job is what’s fixed and the model is what’s swappable, that decision stays sound even as the models underneath it turn over.

That decision is exactly what the harness makes, automatically, on every single request — reaching for the cheapest model that has proven it clears the bar for the capability in front of it. The public bench is the dated readout of those decisions; the machine that makes them, task by task, is what this series takes apart. How it makes that choice in the moment, escalates when confidence dips, and stays honest about cost is the next component we open.

Move the quality slider on the live bench and watch the cheapest qualifier change task by task — the per-job view in action. Next in the series: picking the model at runtime, where the atom we just defined becomes a live decision. Curious how the bar itself gets set? That’s the methodology.