Inference is the line item nobody owns
The cost of running a model in production is not a price you negotiate with a provider. It is a utilisation number your own architecture sets, and almost all of it is decided by four choices made before anyone reads a rate card.
The finance business partner has found the number and wants to know who signed for it. The model spend was a rounding error in the pilot and it is not a rounding error now. Somebody produces the provider’s rate card, somebody else suggests moving to a cheaper model, and the conversation becomes a procurement conversation within about four minutes.
It is the wrong conversation. The rate card is the least interesting variable in the bill, and switching to a cheaper model is the intervention with the worst ratio of disruption to saving available. The reason is structural: what an organisation pays to run a model is a function of how much work it asks for and how efficiently that work is packed, and both of those are properties of the architecture rather than of the contract.
The claim is narrow and it holds in every estate we have looked at. Inference cost is a design output, not a purchasing input. If nobody owns the design, nobody owns the cost — and the line item grows until someone notices it, which is usually a quarter after it stopped being cheap to change.
Where the money is actually decided
Four decisions set the bill, and none of them is made by the person who eventually has to explain it.
- Decision 01 How much context is sent Retrieval depth, chat history, system prompt, tool schemas. Charged on every call, forever.
- Decision 02 How many calls a task takes A single completion, a chain of five, or an agent loop with no fixed ceiling.
- Decision 03 Which model handles which request One model for everything, or a route that sends the easy majority somewhere cheaper.
- Decision 04 How the work is packed Self-hosted only: batching and cache reuse decide how much of the hardware you are paying for is producing tokens.
The first two are where most of the money goes, and they are the two least visible. A retrieval step that returns twelve passages instead of five is a sensible-looking default that raises the input cost of every request the system will ever serve. An agent loop with no step limit is a design that has declined to state its own worst case. Neither shows up in a review of the model choice, because neither is about the model.
The third is the one teams reach for last and should reach for first. Most production traffic is not hard. Classification, extraction, routing and short-form drafting are handled by models an order of magnitude cheaper than the frontier tier, and the published per-token prices across the market span roughly three orders of magnitude from the smallest hosted models to the largest — the Artificial Analysis price comparison is the cleanest public view of that spread. A system that sends everything to the most capable model available is not buying capability. It is buying capability for the eighty per cent of requests that did not need it.
The fourth only applies if you run your own hardware, and then it dominates everything else. Cost per token on self-hosted infrastructure is the hourly cost of the accelerator divided by the tokens it actually produced in that hour. The price of the hardware is fixed. Utilisation is not, and continuous batching, cache reuse and sensible admission control are the difference between an expensive cluster and a cluster that is mostly idle and still expensive.
The version of this that reaches a board
The reason inference spend surprises people is that it is described in the wrong units. A rate card is priced per million tokens. A business is run per case, per claim, per ticket, per document. Nobody has translated one into the other, so the number arrives as an infrastructure overrun rather than as a unit cost that went up.
| What the invoice says | What the business needs to know |
|---|---|
| Spend per month, by provider | Cost per completed case, and how it moved this quarter |
| Tokens consumed | Tokens per case, split into context, output and retries |
| Model tier in use | Share of traffic that genuinely required that tier |
| Total cost of the AI programme | Cost of the thing it replaced, measured the same way |
The right-hand column is not harder to produce. It is a tagging decision and a small amount of arithmetic, and it is almost never made at the start because at the start the number is too small to bother with. That is exactly when it is cheap to instrument, and it is the same argument as versioning a retrieval index before anybody asks a question about it.
What to do before the next renewal
Then make three decisions, in this order. Cap the loop: every agentic task gets a maximum number of steps and a maximum spend, enforced in code rather than in a policy document. Route the traffic: measure what proportion of requests a smaller model handles at acceptable quality on your own evaluation set, and move them. Trim the context last, because it is the change most likely to cost you accuracy and the one you want to make with evidence rather than with a hunch.
There is a fourth thing, and it is the one that pays for the others: decide whether the workload should be inference at all. A statistical baseline still answers a great many enterprise prediction problems at a cost close to zero, and the honest version of an efficiency programme starts by asking which tasks are paying frontier prices for a job a regression did in 2015.
What this does not tell you
None of this is a forecast of what any organisation will pay. We publish no benchmark cost per case, because the number is entirely a function of the estate it applies to, and a figure quoted without that context becomes a target that somebody hits by degrading the system.
It also does not tell you that cheaper is better. The point of measuring cost per case is to be able to spend deliberately — to choose the expensive model for the decisions that carry consequence, having established that the cheap one is sufficient everywhere else. An efficiency exercise that cannot tell those two groups of requests apart is a quality reduction with a spreadsheet attached.
The person who should read this is the one who will be asked, at the next planning round, whether the AI programme is affordable at ten times its current volume. That question has an answer, and it is knowable now. It is a design review, not a negotiation — and the answer stops being cheap to change on the day the first system reaches production without a single one of these numbers being recorded.