Placing a Brand-New Model on the Map (8 of 11)

A frontier model ships on a Tuesday. By Friday, three vendors have blog posts, two have leaderboard updates, and your team is asking the only question that matters: is it good enough for the work we actually run, and is it cheaper than what we use now?

A leaderboard rank won’t answer that. Neither will the model card. You need the new model placed on the same 0–10 scale as everything else, per task, with the same honest confidence bands — and you need it before the news cycle moves on. That’s the problem this post is about: how a model goes from registered to placed on the map quickly, without waiting weeks for traffic to accumulate and without setting fire to the evaluation budget.

This is the harness question in its purest form. A harness is the complete task-specific infrastructure built around a language model to make it reliably do a defined job — and inside a mature harness the model is an interchangeable part. Onboarding a new model is the moment that interchangeability is put to the test: can the harness take a part it has never seen, slot it in, and know within days whether it’s a better fit for the work than the part already there? This post is about the two components that make that possible — the evaluation machinery (verifiers and judges) and the continuous-improvement loop that absorbs a moving frontier week after week.

If you haven’t read the architecture overview yet, start there. This is the closer of the Evaluation arc, and it leans on two of its siblings: how we measure quality and confidence, not just scores. Here we put those mechanisms to work on a model that, an hour ago, the harness had never seen.

See it live: the bench is a dated snapshot — watch a new model appear and its confidence bands tighten from one week to the next.

Why “just wait for traffic” doesn’t work

The honest way to score a model is to watch what it does on real work. That’s the whole premise of the bench: the numbers come from production traffic — the harness running messy, single-pass jobs all day — not a lab harness.

That distinction is the hinge of this whole post, so it’s worth pinning down. A lab harness is a fixed set of curated prompts, run once, scored, ranked — useful for mapping the frontier, but living squarely in the ecological-validity gap between the test and the job. A production harness — ours — is the real, continuous, task-specific machine that runs the actual work. The bench you see is the dated readout of that machine, not a bench-marking rig we stood up to grade models. So when a new model arrives, we don’t score it on invented tests. We score it on what the harness genuinely does.

But real traffic is slow. A brand-new model has done zero work for us. To place it on every task with a tight enough band to trust, we’d need to route a meaningful share of live work to it and wait — days, maybe weeks per task — while judgments trickle in. By the time the model had earned a verdict, it would be two versions old.

The naive fix is worse: judge everything, immediately, at full volume. That places the model fast but blows the evaluation budget, because judging is itself an expensive model call. Do that for every new release across thousands of capabilities and the cost of measuring quality dwarfs the cost of the work being measured.

So neither extreme works. The harness needs to absorb a fast-moving frontier quickly and cheaply and without cutting corners on certainty. Three constraints that usually trade off against each other. The way out is to separate two jobs that look the same but aren’t: placing a new part, and maintaining the parts already on the map.

Replay: real work, on demand

To place a new model fast, we don’t wait for new work. We re-run old work.

The harness has a long memory of real production calls — the exact inputs, the exact prompts, the exact task each one served. To introduce a new model, we take a curated set of those past work packages and replay them through the newcomer: same input, same prompt, fresh model. Then we judge the outputs.

This is the key move. It gives us, on demand, a batch of representative real work to score the model on — not synthetic test prompts written to flatter or trap it, but the actual messy single-pass jobs the harness runs all day. A relevance call that ran last week against a real article. A field-extraction job on a real filing. A summary of a real document. The new model faces the same conditions as the incumbents, immediately, without anyone routing live traffic its way.

Replay is why a model can appear on the bench across every task within a single weekly cycle. We’re not waiting for the world to send us work the model can be scored on. We already have the work. We just run it again — this time with a different part in the socket.

Waves, not one big job

Replaying a curated set across thousands of capabilities is a lot of calls. We don’t fire them all at once and block on the result.

Introduction runs in waves. Dispatch a batch of replays, wait for that batch to come back and get judged, then advance to the next wave. It’s a queue-and-poll rhythm, not a single long-running job that either finishes or hangs.

That design choice is doing real work. New providers are often the least reliable — fresh capacity, rate limits, occasional timeouts. If onboarding were one monolithic blocking job, a single slow provider could wedge the whole thing. Instead, each wave fully drains before the next begins, so a stall in one batch is contained: the process waits, the batch eventually completes or fails cleanly, and onboarding moves on. A flaky launch-week model can’t take the introduction pipeline down with it. Placing a new part must never be able to jam the machine that placed it.

It also means progress is visible and resumable. You can watch a model climb onto task after task, wave by wave, rather than staring at a black box until it either appears fully-formed or doesn’t.

Forced judging: making the band real

Here’s the part that separates “the model showed up on the bench” from “the harness actually knows where it stands.”

In normal operation the harness judges only a sample of calls — more on that in a moment. But during introduction, every replayed output is judged. No sampling, no skipping.

We force it for a concrete reason. A model’s place on the bench isn’t a bare score; it’s a score plus a confidence band — how sure we are. Bands tighten only as judgments accumulate. If we sampled lightly during introduction, the new model’s interval would stay stubbornly wide, the onboarding would stall short of a usable band, and the model would sit in limbo: technically present, practically unverdicted.

Forcing a judgment on every replay is what drives the interval down fast enough to matter. A new model typically lands in the medium or high bands within its first snapshot — present, scored, usable, but openly provisional — and then tightens toward ranked as the judged replays pile up and, later, as organic traffic confirms the picture. The judging that powers all of this is the same panel machinery described in who watches the judges: independent graders, cross-provider exclusion, continuous calibration. Introduction doesn’t get a cheaper, looser judge. It gets the harness’s real evaluation component, on every call — which is exactly why the resulting band is trustworthy.

Deterministic sampling: the affordable steady state

Forced judging is the right call for a fresh model with no history. It would be ruinous as a permanent policy. Once a model is on the map, the harness cannot afford to judge every one of its thousands of daily production calls. So in steady state, it samples — and the sampling has two properties that make it both cheap and honest.

First, it’s deterministic. The decision to judge a given call isn’t a coin flip. It’s derived from a stable fingerprint of the call itself, so it’s reproducible and the realized sample rate actually matches the target rate. No drift, no clustering, no “we meant to judge 2% and somehow judged 9% of one task and 0.3% of another.” You set a rate; you get that rate.

Second, it’s band-aware — as a two-speed switch, not a smooth dial. The state that drives it is the LOW band: the widest, least-certain one, where a pairing is brand-new or barely judged. While a pairing sits at LOW, the harness judges it aggressively — effectively every call — to pull the interval in fast (up to a spend cap, so even a stubbornly-uncertain pairing can’t bill forever). The moment it clears LOW, judging drops to a thin fixed trickle — a fraction of a percent of calls — just enough to catch silent regressions if a provider quietly swaps the model underneath us. And once a pairing has accumulated plenty of evidence, judging stops altogether; there’s nothing left to confirm. Note what this isn’t: the rate doesn’t glide down in step with the band — medium, high, and ranked all sit at that same thin trickle. The switch is LOW versus not-LOW, plus a full stop once the evidence is sufficient. Spend the judging budget where the uncertainty is; stop paying to re-confirm what’s already settled.

That’s the whole trick of the steady state: certainty is expensive, so buy it where it’s missing and stop buying it where it isn’t. Forced judging during introduction and band-aware sampling afterward are the same principle applied at two different moments — spend hard on confidence while you lack it, then stop once you have it. Together they are the continuous-improvement loop that keeps the harness’s part inventory current: every new frontier model gets placed, priced against the incumbent, and either earns a socket or doesn’t — cheaply enough to do it again next week.

You can watch it happen

Because the bench publishes as dated, immutable snapshots — a frozen, reproducible photograph of where every model stood on a given day — the appearance of a new model isn’t a quiet edit. It’s a visible event. And because each snapshot is produced by holding prompts, validation, evaluation, and cost constant and letting only the model vary, the arrival is a controlled experiment: the harness is fixed, the part changed, and the readout shows exactly what that swap bought.

Pull up the heatmap one week and the model isn’t there. Pull up the next snapshot and it is: scored across every task, sitting in medium or high bands, perhaps already wearing the ★ as the cheapest qualifier on a task or two where it’s both good enough and cheap. Walk forward through subsequent snapshots and watch its bands tighten as the judged replays and real traffic accumulate. The bench doesn’t just tell you a model arrived — it lets you replay the arrival, week by week, and see the certainty firm up. (Curious how a number earns a band in the first place? The methodology lays it out.)

This is what lets the harness keep pace with a frontier that ships something new most weeks: replay places a model quickly, forced judging makes its band real instead of merely present, and deterministic sampling keeps the ongoing cost of certainty in check. A harness that couldn’t do all three would either lag the market or go broke trying not to.

There’s one cost we’ve kept waving at and never opened up: the price on the bench. We’ve said it’s not the rate card, that it’s reconciled against the invoice, that it’s blended across failures and retries. Next we start the Billing arc and make good on that — beginning with the deceptively hard question of what a token really costs.

Open the heatmap, find the newest model, and check its bands — then come back next snapshot and watch them tighten. Subscribe to catch the Billing arc as it lands.