Two Honest Prices and a Paper Trail (4 of 11)

When you price a model, you usually copy one number off the provider’s page and move on. But that number assumes you need the answer now. A surprising amount of enterprise work doesn’t: scoring a backlog of support tickets, summarizing yesterday’s filings overnight, classifying a queue that nobody is watching in real time. For that work, the same model — the same weights, same quality — is often available at roughly half the price. You just have to be willing to wait.

That’s one of two ideas this post is about, and they turn out to be the same idea wearing different clothes. Both are parts of the harness — the task-specific machine that surrounds a model to make it do a defined job reliably. The first is honesty about how the work ran: the lane it took, which is the harness’s cost-measurement and recovery machinery. The second is honesty about what was asked: the exact prompt behind every answer, which is the harness’s prompt architecture and provenance. Both close out the inference arc — see Part 2 for the overview, and the siblings on the unit we measure, runtime routing, and engineered reliability. Cost and trust, it turns out, both come down to the harness being honest about its inputs.

See it live: flip the cost mode between batch and sync on the heatmap — same models, two honest prices. Every prompt behind the scores is published on the methodology page.

The patient lane

Most major providers run two lanes for the same model. The sync lane gives you an answer in seconds — that’s what an interactive product needs. The batch lane takes your work, processes it whenever there’s capacity, and hands results back later. Not instantly. Later — sometimes minutes, sometimes hours.

In exchange for that patience, the discount is steep. It’s commonly around 50% off. Identical model, identical output quality, half the cost. The only thing you trade is the expectation of an immediate reply.

For a lot of work packages, that’s a trade you’d take every time. A nightly report doesn’t care whether each summary came back in two seconds or two hours, as long as it’s all done by morning. Scoring a month-old backlog has no clock on it at all. If half your real workload is latency-tolerant — and for most enterprises a large slice is — then quoting only the instant price overstates what that work actually costs, sometimes by a factor of two. A harness built around a task, not a model, has to know which lane the task belongs in — because the lane is half the price.

Same models, two honest prices

So the bench refuses to pick one. On the home page there’s a “batch if supported” toggle, and it does exactly what it says.

Flip it on, and the blended cost for each model uses that model’s batch price wherever the model offers a batch lane — and quietly falls back to its sync price where it doesn’t. Flip it off, and every number uses the instant price. Same models, same quality bar, two honest prices. You pick the one that matches your latency tolerance, and the rankings re-sort accordingly.

This matters more than it sounds, because the discount isn’t uniform. Not every model offers a batch lane. The ones that do don’t all discount the same way. So flipping the toggle doesn’t just shave a flat percentage off the board — it reorders it. A model that looks middling at its sync price can become the cheapest qualifier (★) once you’re willing to wait, while another barely moves. The cheapest-good-enough model for your workload depends on a question the provider’s price page never asks: how patient can you be? Inside the harness the model is an interchangeable part; the lane is a property of the task — and it’s the lane, not the model, that decides half the bill.

The frontier-versus-efficient gap we keep returning to — roughly $15-26 per million tokens at the top versus $0.25-1.50 for the efficient tier, a 30-60x spread — widens in the batch lane, because the discount compounds with the choice. Patience and the right model selection multiply.

What happens under the hood

The toggle is a pricing view, but it’s backed by a real mechanism inside the harness, and the mechanism is the interesting part.

When the harness runs eligible work, it submits it as batches, polls for the results to come back, and then routes each finished item back to the exact job that asked for it. That last step is the part people underestimate. A batch of ten thousand requests comes back as ten thousand results, and every one has to land in the right place. Get that wiring wrong and you’ve got a fast, cheap way to corrupt your own data.

It’s also self-healing. If a batched item fails outright, the harness redispatches the work — to the instant lane, or to a different model entirely — so nothing sits in limbo forever waiting for a reply that isn’t coming.

The subtler safeguard is a deadline on patience. “Later” is fine; “whenever — maybe never” is not, and a batch lane can stall when a provider is having a bad day. So every request carries a limit on how long it’s willing to wait, chosen by the caller and capped by the provider’s own maximum. Most work is genuinely patient — an hour is usually fine, often more. But if the results aren’t back by that deadline, the harness stops waiting: it pulls the unfinished work out of the batch and re-sends it to the next option, whether that’s the instant lane or a different model. That one rule is what keeps a cheap lane from turning a provider’s bad day into a full-day delay. An hour of waiting is an easy trade; a silent twenty-four is not — so the discount is real, but the caller sets the ceiling on how long chasing it can cost. This is the harness’s recovery machinery doing its quiet job: a cheaper lane that can never strand the work.

There’s one caveat worth stating plainly, because it’s exactly the kind of thing a price page won’t tell you: not every model that advertises a batch lane actually batches every kind of request. A model can claim batch support and then quietly decline certain request shapes. When that happens, the work falls back to the instant lane and finishes anyway — but it costs the instant price, not the batch price. So the bench reflects observed behavior, not the marketing. If a model’s batch lane doesn’t work for the shape of work in a category, that’s what the number shows. The price page is a promise; the bench is a measurement — the readout of what the harness actually saw.

A score with no prompt is just a vibe

Now the second half — and the second harness component this post is about. Suppose the bench tells you a given model scores 7.8 out of 10 on a task. Useful — but only if you can answer one question: 7.8 on what, exactly?

Phrase a task one way and a model sails through. Phrase it another and the same model stumbles. The prompt isn’t a footnote to the result; it largely is the experiment. Inside the harness the prompt is the task-specific part — the piece that stays fixed while the model varies. A quality number floating free of the prompt that produced it is unfalsifiable — you can’t check it, can’t reproduce it, can’t argue with it. It’s a vibe with a decimal point.

So the harness treats prompts as data, not as throwaway strings. The exact prompt behind every result is stored and versioned, and every recorded call is pinned to the specific prompt version used to generate it. Then the bench publishes them: for each capability you can see the prompt, the grading rubric, and a hand-curated example input and output. You can read precisely what was asked, see what a good answer looks like, and reproduce the test yourself. The methodology page lays out the rules; the published prompts make the scores checkable. This is where the harness and its readout separate cleanly: the versioned prompt architecture is machinery inside the harness; the published prompt is the window the bench opens onto it.

No silent drift

Pinning prompts to results buys something subtle but load-bearing: provenance.

Prompts get improved over time — that’s healthy; a harness that never improves its own prompts is a dead one. The risk is that improving a prompt silently re-grades history, so a model’s number shifts under you with no explanation, and last quarter’s ranking no longer matches this quarter’s for reasons nobody can point to. That’s how a benchmark loses trust without anyone lying.

Because every call is pinned to the prompt version that produced it, that can’t happen here. Improve a prompt and the old results stay attributed to the old version; new results attach to the new one. A model’s score always traces back to a known, named prompt. Nothing shifts retroactively. When a number changes, there’s a reason you can find — a new prompt version, a new run — not a quiet edit to the past. The bench is a dated snapshot of the harness precisely because provenance makes every score in it point at the exact configuration that produced it.

This is the same honesty as the batch toggle, pointed at a different input. There, the harness is honest about the lane the work ran in. Here, it’s honest about the prompt that produced the answer. In both cases the bench earns trust by refusing to hide the thing that actually drives the result.

Closing the loop

Step back and the inference arc holds together — and so does the harness it describes. We defined the unit — the work package, not the model. The harness picks a model for each one at runtime, engineers the call to be reliable, runs it in whichever lane fits the patience the task allows, and pins every result to a prompt you can read. Cost is honest because the lane is visible. Trust is honest because the prompt is visible. And because both are properties of the task rather than the model, swapping one model for another changes a row of configuration and nothing else — the machine stays put.

Which sets up the question the whole second arc exists to answer. The harness can now run work reliably and reproducibly — but reproducible isn’t the same as good. A pinned prompt and a known lane tell you the result is real; they don’t tell you the answer was any good. How do you put a trustworthy number — a quality score you’d actually stake a decision on — on the output of a system that has no single right answer? That is the part of the harness that makes every “good enough” pick above honest, and it is the whole subject of the second arc.

Curious which models win once you can be patient? Flip the batch toggle on the heatmap and watch the board re-sort. Next up, the question that’s harder than it looks: measuring quality — how a 7.8 gets earned.