Routing to "Good Enough": How the Right Model Gets Picked (2 of 11)

A request comes in: score these two hundred articles for relevance. Somewhere a decision gets made about which model runs it — and that single decision, repeated millions of times, is the entire difference between an AI product that’s economically viable and one that quietly bleeds money.

Most systems make that decision once, in a config file, months ago: use the big model, it’s safest. The harness makes it fresh, per work package, every time. And it almost never reaches for the strongest model. It reaches for the cheapest one that has proven it can do this exact job well enough.

That’s routing — the harness component that picks the part. It’s the engine behind every cell on the heatmap, and it’s the concrete, in-code version of what Part 2 called picking the model at runtime. This post opens the box. Routing is only possible because the harness holds everything else — the prompts, the validation, the evaluation, the cost accounting — fixed, and lets a single thing vary: the model. The model is an interchangeable part. Swapping one for another changes a row of configuration, nothing else. Routing is the machinery that decides which part goes in.

See it live: the heatmap is this routing decision drawn out — each ★ is the cheapest model proven good enough for its category. Drag the slider and watch the picks shift.

The decision in one sentence

For each work package, select the cheapest model that has demonstrated it clears that capability’s quality bar — not the most capable model available.

Read that again, because it inverts how almost everyone shops for models. The usual instinct is to find the best model and then look for places it’s overkill. The harness does the opposite: it assumes good enough is the target, and lets cost break the tie among everything that qualifies.

If you’ve read why we benchmark work packages, you know why this only works at the capability level. “Is this model good?” has no answer. “Does this model reliably extract payment terms as clean structured data, and what does that cost?” has a very specific one — and that’s the question routing actually asks, fresh, every call. The capability is the task; the model is whatever the harness plugs into it this time.

Two gates: eligibility, then cost

The selection runs as two gates in order. A model has to pass both.

Gate one is eligibility. To be considered for normal routing, a model must have earned at least a trusted confidence band on this specific capability — medium or better. Not “we tried it once and it looked fine.” We mean its track record on this exact job is tight enough that the harness would stake real work on it. Models that are unproven, or whose quality is still uncertain, are simply not in the running. They’re the greyed-out cells on the heatmap: present, but not yet eligible to win here.

This is the part people miss. You cannot route on quality unless you know, with measured confidence, where each quality bar sits and who clears it. Eligibility is where all the measurement the rest of the harness performs gets cashed in. Everything the evaluation side does — judging, calibration, confidence intervals — exists so that this gate can be trusted.

Gate two is cost. Among the models that passed gate one, rank by true cost and take the cheapest. That winner is the model wearing the ★ on the heatmap: the cheapest qualifier for that capability. Every other eligible model’s cell shows its cost as a multiple of that cheapest qualifier — 1.4×, 3×, 12×. The ★ is at 1.0× by definition. The grey cells aren’t in the comparison at all, because they didn’t clear gate one.

So the heatmap isn’t a leaderboard with prices bolted on. It’s a literal picture of this two-gate decision, run across every model and every category at once — the visible readout of the harness’s selection logic, frozen into a dated snapshot. Part 1’s “portfolios of models matched to portfolios of tasks” — that’s what you’re looking at.

There are other strategies the harness can run — optimize for raw quality, or speed, or a specific blend. But the default that powers the public bench is the one that matters here: cheapest that qualifies.

“Cost” here means the real cost, not the rate card

Gate two ranks on cost, and the cost it uses is not the number on the provider’s pricing page.

A rate card is a per-token sticker price. It tells you almost nothing about what a task costs, because tasks differ in shape. A relevance score reads a long article and emits a tiny verdict; a summary reads a document and writes several paragraphs. Same rate card, wildly different bills. So the ranking uses each model’s measured input-to-output token shape for this capability — taken from real traffic through the harness, not a guess.

Then there’s the gap between list price and the invoice. Failed calls that had to be retried, batch discounts, the per-provider accounting quirks that make “a million tokens” mean subtly different things — none of that is on the rate card. The harness closes that gap by reconciling recorded costs against what providers actually billed, and folding the result into one comparable number. That’s the blended cost the heatmap ranks on, and it’s anchored to money that actually changed hands.

The upshot: a model that’s cheap per token but fails one call in five is not cheap, and the cost gate sees through it. The ★ is a defensible recommendation precisely because it isn’t a sticker price.

Cascading: taking the saving without taking the risk

Routing to the cheapest qualifier buys the savings. But “cheapest qualifier” is only safe if the harness can catch the moments it falls short. That’s cascading — and there are two distinct behaviors, for two distinct situations.

Situation one: a model that’s eligible still produces a bad output on this particular call. Even a proven model misfires sometimes — malformed structured output, a failed objective check, a result that trips a deterministic verifier. When that happens, the harness doesn’t shrug and pass the bad answer downstream. It excludes the offending model and re-runs the work on another, escalating until the output passes its checks or the options are exhausted. A failed call becomes a fresh attempt on a different part, automatically. What makes any of this recoverable in the first place — how a malformed answer gets detected and retried cleanly — is its own subject in engineered reliability.

Situation two: a model isn’t proven yet, so the harness can’t trust it on price at all. This is the other side of the eligibility gate. When a model’s quality on a capability is still uncertain, the harness won’t quietly route to it just because it’s cheap. Instead it fans the work out across several models at once and lets a judge panel pick the winner. The cheap model has to win on quality, repeatedly, before its confidence band tightens enough to make it eligible — and only then can it start winning on price. Confidence is earned before it’s trusted. That fan-out-and-judge process is exactly how a model goes from registered to placed on the scale; onboarding a new model walks through it, and who watches the judges covers the panel that does the picking.

Two behaviors, one principle: routing chases the cheapest answer, and cascading makes sure cheap never means careless.

The same cheap model isn’t always the same cheap model

One more layer, because real work isn’t uniform. The best cheap model for legal relevance scoring may not be the best cheap model for biomedical relevance scoring, even though it’s nominally the same capability.

So selection can stratify by sub-context — by domain, for instance. A model proves itself within a sub-context, and routing prefers the cheapest qualifier for that slice. When there isn’t enough signal yet to judge a slice on its own, selection falls back to the aggregate track record rather than guessing. The savings get sharper as the evidence accumulates, without ever routing blind. The task stays fixed; the harness just gets more precise about which part fits which cut of it.

Why the slider works

Here’s the thing the whole harness earns you. Because every eligible model carries a measured quality score with real confidence behind it, the bar isn’t fixed in concrete — you can move it.

That’s what the quality slider on the bench does. Drag it to “at least 90% as good as the best” and you’re re-running gate one live: the eligible set shrinks, the ★ may jump to a different, slightly pricier model, and the whole heatmap reshuffles in front of you. Drag it down toward “good enough is plenty” and cheaper qualifiers light up. You’re re-asking the core question — which model is good enough for my bar, at the lowest cost? — and getting the answer recomputed in real time.

You can only offer that slider if you know exactly where each quality bar sits. Which is the whole point: routing is the easy part once the measuring is honest. The slider isn’t a toy on top of the readout — it’s the harness’s selection logic exposed, driven live against the same measured confidence the routing itself runs on.

Open the heatmap and move the slider — every ★ you see is this decision, made and re-made. Next we go under routing itself: engineered reliability — what actually makes a failed call recoverable, and why a success rate is a built thing, not a given.