Make model routing earn its own cost
Cheaper small models change the economics of deciding which model to call. Compare direct calls, upfront routing and sequential escalation, including the cost of selecting and checking each answer.
Keep development and test groups separate before comparing performance. No measured results are shown.
On 7 October, Anthropic released Claude Haiku 5.5, a small model aimed at frequent, bounded work such as summarisation, classification and subagent tasks. Its launch prices for prompts up to 100,000 tokens were US$0.10 per million input tokens and US$0.50 per million output tokens. Above that prompt threshold, the published rates were US$0.50 and US$2.50 respectively. Those are provider terms for specified request sizes, rather than a price for completing a business task.
Lower inference prices make more routine work worth testing. They also change the relative cost of deciding which model should handle it. A separate classifier, checking model or sequential fallback can consume a material part of the budget for an otherwise inexpensive response. The architecture owner should compare the routing mechanism with direct calls before adopting it.
The useful question is specific: does selecting between models improve the quality and cost combination enough to pay for the selection itself?
Distinguish selection from sequential escalation
An upfront router chooses a model before generating the answer. It might use a task rule, a trained classifier or another language model. A cascade starts with a cheaper model, checks its answer and calls a stronger model when the check does not accept it. The latter buys evidence about the first answer, but pays for that answer even when it is discarded.
RouteLLM, first published in June 2024 and revised in February 2025, distinguishes these mechanisms. Its learned routers use preference data to select one responding model per query. Its experiments separately report routing overhead and show that performance depends on the relationship between training and evaluation queries. This is an older research foundation, not a test of Haiku 5.5 or an enterprise service.
For a hypothetical document operation, an upfront rule could send short field-extraction requests to the small model and complex cross-document analysis to a stronger one. A cascade could instead attempt every extraction cheaply, check whether required fields have source support and escalate unresolved cases. These designs need different records and have different latency. A successful second answer does not remove the first call from the bill.
The direct small-model route belongs in the comparison too. If it already meets the bounded task’s acceptance conditions, a router must justify why another component is needed. If it does not, direct use of the stronger model supplies another baseline. An existing task category may provide a simpler selection rule than a learned router.
Calculate the cost of the route you are proposing
Consider a constructed example, with all variable costs in pence per incoming request. Assume a small-model answer costs 10 pence, a stronger-model answer 50 pence and either an upfront selection or a post-answer check five pence. These are invented amounts for inspecting the mechanism. They are not Haiku prices or converted provider rates.
| Constructed example | Pence per incoming request |
|---|---|
| Direct small model | 10, if its output meets the requirement |
| Direct stronger model | 50 |
| Upfront router sends 75 per cent to the small model | 5 + 0.75 × 10 + 0.25 × 50 = 25 |
| Small model first, then 40 per cent escalate | 10 + 5 + 0.40 × 50 = 35 |
The different traffic shares in this constructed example are assumptions, not a measured comparison. The table cannot rank quality. It shows which calls each design purchases. With these particular amounts, a cascade reaches the direct stronger-model cost when 70 per cent of requests escalate. Above that share, its variable inference cost exceeds the direct route. A different checking cost, answer length or retry policy moves that boundary.
Fixed overhead can reverse an apparently favourable comparison at low volume. In the same illustrative example, suppose operating the upfront router also costs 200 pounds per month. Against 50 pence per direct request, the router breaks even at 800 requests per month: each route then costs 400 pounds. At 10,000 requests, the constructed totals are 2,700 and 5,000 pounds. Both exclude downstream review and correction at this stage.
Find the volume that pays for the router
Swipe or scroll for the full diagram →
- Fixed assumptions · monthly GBP
- Fixed £200.00; model £0.25 per decision; loaded labour £30.00 per hour; rework share 0, cost £0.00 per reworked decision.
- Matched incumbent scenario
- £5,000.00 per month at the same volume; £0.56 per accepted decision at the same assumed acceptance share. Incumbent unit cost: £0.50.
- Accounting boundary
- Review is the initial human check. Rework is a separate correction after that check; its unit cost excludes the initial review labour. Both can occur on the same case.
- Formula and limit
- Monthly cost = fixed + volume × model unit cost + volume × review share × review minutes ÷ 60 × hourly labour + volume × rework share × rework unit cost. Divide by volume × acceptance share. This sensitivity model establishes neither causality, realised savings nor a staffing reduction.
Constructed monthly GBP assumptions, not Anthropic prices, a currency conversion or measured savings. Review and rework start at zero to isolate routing economics. The incumbent uses the same volume and assumed acceptance share.
A hypothetical router costs an additional 200 pounds per month and 25 pence per incoming request, compared with 50 pence for a direct route. Both are assumed to accept 90 per cent of requests. At 800 requests their monthly costs are equal. Change volume to see how the fixed cost affects the comparison.
Reviewed 2026-10-14
This illustrative calculation holds the variable routed cost at 25 pence and starts with equal assumed acceptance of 90 per cent. Change volume to inspect the fixed overhead. Changing acceptance applies the same share to both sides, so this interaction cannot represent a quality difference between routes. Human review can be added to the candidate as a sensitivity check. A real comparison needs each route’s own measured acceptance, review and rework.
Reprice the calls and preserve the configuration
Do not multiply an old token count by a new rate without checking the request. Anthropic’s launch footnote says Haiku 5.5 uses an updated tokenizer and slightly more tokens for equivalent work. The release also introduces an adjustable effort setting. Input size, generated length, effort, cache use and retries therefore belong with the recorded model version.
The system card’s capability summary, Table 8.1.A on page 111, says its Haiku 5.5 results normally use adaptive thinking at maximum effort and average five trials. A budget trial at another effort setting is a different configuration. The reported benchmark results do not establish its local acceptance or escalation rate.
RouteLLM’s cost and overhead analysis used its own model pair, token-price assumptions and serving hardware. Reuse the distinction between generation and selection cost. Recompute their contributions for the proposed workload rather than importing the paper’s savings ratio.
Choose a route on held-out work
Take one bounded task with an agreed acceptance condition. Compare direct small and stronger models with a simple upfront rule and, where useful, a cascade. Tune selection thresholds separately from the held-out comparison. Preserve the same source access and task requirements across routes, and review outputs without revealing which route produced them.
Record every call against the original request, including selection, checking, discarded answers, retries and escalation. Measure final accepted work, total cost, time to a usable result and human correction by task type. Inspect cases accepted by the cheap route that should have escalated. A low escalation rate can mean efficient coverage or a check that misses errors.
The earlier inference-cost article establishes the unit of accepted work. This decision sits inside that accounting boundary: whether to buy an additional selection mechanism at all. The architecture owner should retain the simplest route that meets the task requirement at an acceptable total cost, and revisit the choice when prices, traffic or model configuration change.