We Don't Trust the Rate Card. We Reconcile Against the Invoice (10 of 11)
Pull up almost any “cheapest model” comparison and you’re reading the same number we are: the price page. List rate, times tokens, done. It’s the input everyone shares, and it has a quiet flaw — nobody checks it against the bill.
Our harness does. On this bench, the cost next to every model isn’t the rate card. It’s a reconciled number, anchored to what providers actually charged us, corrected continuously as the real invoices come in. That’s the single biggest reason a cost comparison here is worth trusting over one that stops at the pricing page.
This is the cost-measurement component of the harness — component eleven, the one that keeps every dollar figure honest — and it’s the cost backbone of the whole series. If you haven’t yet, What a token really costs covers the careful version of the list calculation — banded pricing, batch lanes, the per-provider token-counting quirks. This post is what happens after you’ve done all that carefully and the invoice still disagrees. (Part 2 is the map for the whole harness.) And it matters downstream: the corrected cost here is exactly what the harness’s runtime selector spends against when picking the model at runtime.
See it live: the costs on the bench are reconciled actuals, not rate-card math — that’s the whole point of this post, and why the ★ means something.
The rate card is a starting estimate, not the truth
Here’s the uncomfortable part. You can compute the list cost of a call perfectly — exact token counts, the right banded price for the model, the batch discount applied correctly — and the figure can still be materially wrong by the time the bill arrives.
Why? Because a pricing page describes one clean call in isolation, and a production harness never runs one clean call in isolation — it runs messy, single-pass work all day. The gap between “what the price page says this cost” and “what we actually paid” opens up for a pile of mundane reasons:
- Retries. A call fails its check and re-runs on another model. The price page never mentions the second attempt.
- Caching you didn’t anticipate. Some providers discount repeated context automatically. Great for the bill — but your list estimate didn’t know about it, so it’s now too high.
- Batch lanes and tier effects. Work that runs as a batch is cheaper, sometimes by half. Volume tiers and account minimums nudge the real rate further from the sticker.
- Mid-cycle price changes. Providers re-price. The page you read last week is not the page you were billed under.
- Provider-side quirks. Rounding, free-tier crossovers, the occasional SKU that doesn’t map cleanly to anything you recognize.
None of these is exotic. They’re just normal, and they all live in the space between the rate card and the invoice. Treating the rate card as truth means treating that whole space as zero. It isn’t.
What “reconcile against the invoice” actually means
So the harness doesn’t. It records the computed list cost of every single call it makes — the careful estimate, model by model. Then, periodically, it lines those records up against what each provider actually charged, pulled from the billing exports and invoices the providers themselves produce.
The comparison is bucketed: by day, by model, and by lane (batch versus sync). Day, because that’s the grain billing data arrives in. Model, because that’s the unit the bench ranks. Lane, because a batch dollar and a sync dollar are genuinely different prices for the same work and collapsing them would smear the result.
For each bucket the harness now has two numbers sitting side by side: what it computed and what it was billed. The whole game is making those two agree — and when they don’t, understanding why instead of papering over it.
We check it both ways, and we hide nothing
A naive reconciliation only looks one direction: did we overestimate? That misses half the picture. The harness compares the two sides as a union of buckets, which surfaces two different kinds of disagreement — and both are real signals.
We computed a cost the provider never billed. That points back at the harness itself: an over-count, a double-count, a call it recorded but that somehow didn’t show up on the invoice. It’s a bug-on-our-side detector.
The provider billed something we have no matching call for. That’s an unattributed charge — a SKU, a fee, a line item we can’t tie to any call the harness made. This is the one most systems quietly drop on the floor.
The harness doesn’t drop it. Anything it can’t map to a known call rolls into a clearly-flagged catch-all bucket, and the raw provider line is kept verbatim for audit. The principle is simple: nothing silently disappears. If the provider charged for it, it shows up somewhere in the books, even if all we can say about it for now is “we don’t yet know what this was.” An unexplained charge you can see is a problem you can chase. An unexplained charge that’s been rounded away is a problem that quietly corrupts every number downstream.
Billing data lands late — so we wait for it to settle
Here’s a subtlety that took real care to get right, and that a casual comparison would never even notice.
Provider billing data is not final when usage happens. It lands late, and it gets restated for a day or two afterward as the provider’s own accounting catches up. Today’s spend isn’t fully reported yet; yesterday’s might still move a little.
That creates a trap. If you compare your real-time computed cost against billing data that hasn’t finished settling, the most recent days look artificially cheap — you’ve counted all your calls but only some of the provider’s charges have shown up. Average that in and your whole correction gets biased toward “everything is cheaper than it is.”
So the harness self-calibrates. For each provider it finds the most recent fully-settled day, and drops the still-settling tail from both sides before comparing. It doesn’t hard-code “providers are two days behind” — it lets each provider effectively self-report its own settlement lag by watching when its numbers stop moving. One provider might settle in a day; another might take three. The reconciliation adapts per provider instead of guessing.
One honest exception, because it matters for accuracy: a few providers bill by drawing down a prepaid balance rather than sending an itemized invoice. For those, the balance change is the money — there’s no settlement lag to wait out, so there’s nothing to clip. The harness handles that case on its own terms rather than forcing it through the same window.
The correction factor, and why drift is a feature
Out of all this comes one number per model: a correction factor — essentially the smoothed ratio of what we actually spent to what the harness computed as the list cost.
Two design choices keep it honest:
It’s smoothed over time. A single weird day — a usage spike, a one-off billing artifact — shouldn’t whipsaw a model’s cost. The factor tracks the settled trend, not the noise of any one day.
And it’s clamped to sane bounds. If the math ever produces a wildly out-of-range factor — a model that supposedly costs ten times its list price — the harness treats that as a data problem or a bug, not as real overhead. Reality has limits; a number outside them is telling you something broke, not that the model got expensive.
The distance between list and actual is what we call drift — and a large drift is itself one of the most useful signals the whole harness produces. A model whose actual cost suddenly pulls away from its list estimate is telling you something: a price change you hadn’t ingested, a new caching behavior, a retry storm, or a bug worth chasing. Drift isn’t an error to suppress. It’s a smoke alarm. Most cost comparisons can’t even see drift, because they only ever hold one of the two numbers.
The payoff: the corrected cost — not the rate card — is what feeds the bench and what the harness’s runtime selector spends against. The same honest number does double duty: it ranks the models you see, and it’s the budget the harness optimizes against on every live call. This is the task-specific-not-model-specific spine again — the correction factor is a per-model row of configuration; swap one model for another and the harness just starts reconciling a different row.
Why this is the difference between budgeting and being misled
Here’s the practical stakes, in one line: a model can look cheap on the price page and cost materially more in practice — or look expensive and quietly come in under.
If your comparison stops at the rate card, you can’t tell which is which. You’ll pick the model with the prettiest sticker, get billed by reality, and discover the gap on your own invoice three weeks later. At a few hundred calls, that’s an annoyance. At millions of calls — the volume a production harness actually runs — it’s the line between a workload that’s economically viable and one that isn’t.
This is also why the ★ “cheapest qualifier” on the bench means something. Naming the cheapest model that clears a task’s quality bar is only an honest recommendation if “cheapest” is grounded in money that actually changed hands. Reconciled cost is the only defensible basis for that claim. A benchmark that ranks on list price is, at best, ranking on the providers’ marketing copy. That’s the difference between a lab harness — a fixed set of curated prompts, priced on the sticker, run once and ranked — and a production harness that gets billed for its work and checks the bill.
We’d rather rank on the invoice. And because the harness holds prompts, validation, evaluation, and now reconciled cost constant while only the model varies, the bench is the honest readout of it: a dated snapshot of what a stable harness costs when the sticker isn’t allowed to lie.
The corrected, reconciled cost is one ingredient, though — not the final number. A model that’s cheap per token but fails one call in five isn’t actually cheap, and a fair comparison has to fold in reliability and the task’s real token shape too. That’s the last piece of the cost story: blended cost — how the reconciled price, the success rate, and the input-to-output token shape collapse into the single number the bench actually ranks on. That’s next.
The price page is what they want you to compare on. The invoice is what you’ll actually pay. Move the heatmap and the numbers you’re reading are the second kind — settled, corrected, real.