What a Token Really Costs (9 of 11)

This post is about one component of the harness — the machine of task-specific infrastructure a language model runs inside so it reliably does a defined job. The component here is cost measurement: the part of the harness that answers “what did this call actually cost to do the work,” honestly enough that a price ranking can be trusted.

Here’s a calculation that looks airtight: take the number of tokens a call used, multiply by the model’s advertised price per million, and you have what the call cost. Two models, same task, same token counts — cheaper rate wins.

It’s wrong. Not by rounding error — wrong in three structural ways that can flip a cost ranking. The headline rate is the list price of a token in isolation, and almost nothing about a real call happens in isolation. The umbrella post flagged that cost is “surprisingly not ’tokens × rate.’” This post opens the billing arc by showing exactly where that folk model breaks — and why even the careful version isn’t the final truth. The harness has to get this right because every routing decision downstream leans on it: a cost number that lies quietly poisons which model the machine picks.

See it live: the “$ per 1M” figures on the heatmap already bake in the token-accounting quirks this post unpacks — toggle batch vs sync and watch the lane change the math.

The shape of a call, not just its size

Start with the assumption hiding inside “tokens × rate”: that a token costs the same no matter how it arrives.

Some providers price by request size, in tiers. A request up to a certain token threshold bills at one rate; cross that threshold and the whole request prices at a higher band. Not the overage — the entire call. So two workloads that move identical total volume can cost noticeably different amounts depending on how the work was chunked. Send one large request that tips into the upper band, and you pay the upper rate on every token in it. Split the same work into smaller calls that each stay under the line, and you don’t.

The consequence is uncomfortable for anyone who likes tidy arithmetic: cost depends on the shape of your calls, not just the total tokens. “How many tokens this month” doesn’t determine the bill. “How those tokens were packaged into requests” does. A lab harness — a fixed set of curated prompts run once and scored — only ever knows aggregate volume, so it is already blind to this. The production harness measures cost at the level of the individual call, packaged the way the real workload packages it, rather than dividing a monthly total by a token count.

Reading is cheap, writing is dear

The second crack is one most people know about and still under-weight: input and output tokens don’t cost the same.

Output is typically several times more expensive than input. That asymmetry means a task’s true cost is governed by its input-to-output ratio — how much the model reads versus how much it writes. A classification task reads a long document and emits one word; it’s almost all input, so it lives near the cheap end. A drafting task reads a short brief and writes a thousand words; it’s output-heavy, and that’s where the expensive tokens pile up.

This is why a model’s headline “cheap” can be a trap. A model with a low input rate but a habit of verbose, padded output may be genuinely expensive on a write-heavy task — and genuinely cheap on a read-heavy one. There is no single “this model is cheap” answer; there’s only “cheap for this task’s read/write balance.” Which is the same lesson the harness keeps teaching in every component: the unit that matters is the task, not the model. The model is an interchangeable part; the read/write shape belongs to the job it’s plugged into.

Which means you can’t guess the ratio — you have to know it. The harness measures each task’s characteristic input-to-output shape from real production traffic, not from a tidy assumption, because the difference between “reads a lot, writes a little” and the reverse can be the difference between a model qualifying on price and not. That’s a big part of what folds into the blended cost the bench ranks on.

The same word, counted differently

Now the part that trips up even careful teams — and the part that quietly broke our own numbers for a while.

Providers don’t just charge differently. They report usage differently, and if you take every provider’s numbers at face value, you will charge yourself twice for the same tokens without noticing. Two flavors of token make this vivid.

Cached input tokens. When a model reuses a chunk of input it’s seen before, that chunk is usually billed at a discount. But how the provider reports it varies. Some include the cached tokens inside the total input count — so the cached number is a subset, and you have to peel it out and price it separately, or you’ll charge full freight for tokens that were actually discounted. Others report cached tokens separately, already excluded from the main input total — so if you add them back in, you’ve counted them twice.

Reasoning tokens. Same trap, different bucket. Some providers fold the model’s internal “thinking” tokens into the output count; others report them as a separate line to be added on. Whether a kind of token is part of another total or separate from it is a per-provider property — there is no universal convention, and the price page rarely spells it out.

Get this wrong in one direction and you undercount; get it wrong in the other and you inflate. There is exactly one correct accounting per provider, and it has to be maintained inside the harness as a deliberate map: for each provider, is “cached” a subset or separate? Is “reasoning” inside output or added to it? Each kind of token must be billed exactly once.

A bug, told honestly

We learned this the way you usually learn it: by getting it wrong.

For a stretch, the cost component assumed one provider’s convention applied to everyone — that cached and reasoning tokens were always billed as separate additions. For the providers where that’s true, fine. For the providers where those tokens are actually a subset of a total we were already counting, we added them in a second time. The effect was a quiet, one-directional inflation: those models looked more expensive than they really were. Nothing crashed. No error fired. The cost column just lied, consistently, in favor of the models whose convention we happened to match.

The fix wasn’t a clever line of code. It was building the per-provider convention map — encoding, provider by provider, which tokens are subsets and which are separate — and then re-pricing the historical calls so the back-record matched reality too. A correction you don’t apply to history is a correction that leaves your past rankings wrong.

The lesson generalizes past our particular slip, and it’s the same discipline the whole harness runs on: a benchmark that doesn’t model these accounting quirks will get cost rankings systematically wrong, and you would never catch it from the vendor’s price page. The page tells you the rate. It does not tell you whether the usage numbers you’ll be charged against are double-counting themselves. The bench is honest about cost only because the harness underneath it is.

Why the bench bakes this in

All of this is why the “$ per 1M tokens” figures the bench shows are not the numbers off a pricing page. The bench is the readout — a dated snapshot of what the harness’s cost component computes — and those figures are the result of modeling the call shape, respecting the input-to-output asymmetry that the real task exhibits, and applying the correct per-provider token accounting so nothing is counted twice or missed once. The point of a work-package benchmark is to tell you what a model costs to do the job — and “the job” includes all the boring arithmetic the rate card omits. Hold prompts, validation, and evaluation constant, price each call the way the real workload prices it, and let only the model vary: that is what makes the cheapest-qualifier ★ mean something.

If you want the inputs and assumptions laid out, they’re in the methodology; the live rankings are on the heatmap, where the cheapest qualifier on each task carries the ★.

And it’s still not the final truth

Here’s the twist that opens the rest of this arc. Suppose the harness models banded pricing perfectly. Suppose it nails the read/write ratio from real traffic. Suppose its per-provider convention map is flawless and every token is counted exactly once.

You still don’t have the real bill.

Because the rate card doesn’t know about the calls that failed and got retried, the caching discounts you didn’t anticipate, the cheaper batch lanes, the minimum charges, or the price that quietly changed mid-month. Careful per-token math gets the harness a very good estimate of cost. It does not get the invoice. And those two numbers drift — sometimes by enough to matter.

That gap is the subject of the next post. The harness doesn’t just compute what a call should cost; it checks that estimate against what the providers actually billed us — and treats the difference as data, not noise.

Next: we reconcile against the invoice — what happens when your careful estimate meets the statement that actually arrives. Move the slider in the meantime; the cost ranking it reshuffles is the honest one, not the rate card’s.