The cheapest token is the one you did not send
Caching is treated as an optimisation to be added once the bill hurts. It is a design decision that has to be made in the first week, because what a system can cache is determined entirely by how it assembles a prompt.
The efficiency exercise usually starts at the end. Somebody looks at a bill, somebody proposes a cheaper model, and a team spends three weeks re-testing prompts against a smaller model to recover a fraction of what a cache would have recovered in an afternoon.
The reason the cache is not considered first is that nobody built the system so that it could have one. Caching in an inference system is not a component you install. It is a property of the order in which a prompt is assembled — and by the time the bill is worth attacking, that order is baked into a codebase and several thousand lines of prompt.
The claim: prompt structure is a cost decision, and it is made before a single request is served. Systems that put stable material first and variable material last can reuse most of what they send. Systems that interleave the two cannot, at any price.
Four ways not to send a token
- Lever 01 Do not call the model A lookup, a rule, a regular expression or a stored answer. The largest saving available and the one nobody counts.
- Lever 02 Reuse the prefix Provider-side prompt caching charges a reduced rate for a repeated prefix. It requires the prefix to actually repeat, byte for byte.
- Lever 03 Answer from a previous answer A semantic cache over prior questions and responses, with an explicit freshness rule and an explicit blast radius when it is wrong.
- Lever 04 Send a smaller context Fewer retrieved passages, shorter history, a trimmed tool schema. The lever most likely to cost accuracy, so it goes last.
The first is the one that gets missed, because it does not look like AI engineering. A meaningful share of the traffic reaching a production assistant is the same twenty questions, asked in different words, whose answers change monthly. Those do not need a model at inference time. They need a model once, at authoring time, and a retrieval step afterwards.
Why prefix order is the whole game
Provider-side caching works on prefixes. The system compares the beginning of your request against what it has already processed, and charges a lower rate for the portion that matches. The match is exact and it is positional: one changed character early in the prompt invalidates everything after it. Anthropic’s prompt caching documentation is explicit about the consequence — the system prompt, the tool definitions and the leading messages have to be byte-identical across requests, and a minimum prefix length applies before anything is cached at all.
Which means a prompt assembled in this order caches almost completely —
- The system instruction, which never changes
- The tool and schema definitions, which change on release
- The stable policy or reference material, which changes weekly
- The retrieved passages for this request
- The conversation so far
- The user’s current message
— and the same prompt with the user’s name interpolated into the system instruction, or a timestamp at the top, caches nothing at all. Both versions look identical in review. Their bills differ by a large multiple, and the difference compounds on every request the system will ever serve.
This is why it is a first-week decision. Reordering a prompt template after a year of production is not a code change. It is a re-evaluation of every prompt in the system, because moving material changes what the model attends to, and a change that saves money and quietly costs accuracy is worse than the bill.
The cache nobody governs
Semantic caching — answering a new question with a stored answer to a similar one — is the lever with the best economics and the worst failure mode, and it is routinely added without anybody writing down what it is allowed to do.
Two questions have to be answered before it goes in, and they are governance questions rather than engineering ones. First: whose answer is this? A cache keyed only on question text will happily serve one user’s entitlement-filtered answer to another user who was not entitled to it, which is a data breach with a performance improvement attached. The key has to include the entitlement scope, not just the words. Second: how wrong can it be? An answer cached for a day is fine for a policy summary and unacceptable for an account balance, and the freshness rule belongs to the data, not to the cache configuration.
The third line is the one that saves an investigation later. If the record does not distinguish a generated answer from a served one, then the first question after an incident — did the model produce this, or did we hand back something from last Tuesday — has no answer.
What this does not tell you
Caching does not survive contact with genuinely novel work. A system doing research, drafting or open-ended analysis will see a low hit rate and should, and forcing the number up by loosening similarity thresholds is how a cache starts returning confidently wrong answers to questions nobody asked.
It also does not remove the case for a smaller model or for better retrieval. It reorders them. Caching is applied first because it is the only lever on this list that reduces cost without changing what the system produces, which means it can be evaluated on the bill alone. Every other lever has to be evaluated on the bill and on the evaluation set together, and that is a longer piece of work.
The reader who acts on this is whoever is writing the first prompt template for a system that will be in production for years. Put the fixed material first. It costs nothing today, it cannot be done cheaply later, and it is the difference between an efficiency programme that is a configuration change and one that is a project.