What an inference architecture has to hold
Most enterprise AI systems have a serving layer that is really one call to a provider with some retry logic around it. Five things belong in that layer, and the two that are almost never built first are the two that cannot be added afterwards.
Ask to see the architecture and you are usually shown a diagram with three boxes: the application, a vector store, and a rectangle labelled with a provider’s name. It is not a dishonest diagram. It is what the system is. The question it cannot answer is the one that arrives about eight months later, when a second team wants to use the same capability, a regulator wants to know what a particular answer was based on, or somebody wants to move a third of the traffic to a cheaper model without editing forty prompts.
At that point the rectangle turns out to have been load-bearing, and the work that follows is not an upgrade. It is a rebuild with production traffic on it.
The claim is simple. An inference layer is five separable concerns, and the order in which teams build them is almost exactly the reverse of the order in which they become expensive to add.
The five layers
- Layer 01 Gateway Identity, quotas, redaction, refusal policy, and the single point where a request can be stopped.
- Layer 02 Routing Which model serves which request, what happens when it is unavailable, and how a change of provider is made without touching the application.
- Layer 03 Context assembly Retrieval, entitlement filtering, ordering and truncation. What the model is actually given.
- Layer 04 Serving The hosted endpoint or your own accelerators. Batching, caching and the concurrency the system can absorb.
- Layer 05 Record What was asked, what was retrieved, which versions answered, what came back, and what the person did with it.
Most teams build layer four and part of layer three, because that is what makes the demonstration work. Layers one and five are the ones that get deferred, and they are the two that decide whether the system can be operated, audited, migrated or shared.
Why the gateway is not optional infrastructure
A gateway sounds like plumbing and behaves like policy. It is the only place in the system where a rule can be applied once and hold for every application that follows — which is the difference between an organisation that has an AI capability and one that has eleven independent integrations with the same provider.
Concretely, it is where these live: the identity of the calling system and the person behind it, the quota that stops a runaway loop from becoming a bill, the redaction step that keeps categories of data out of a third party’s logs, and the record hook that makes layer five automatic rather than something each team remembers. Without it, every one of those becomes a per-application decision, made under launch pressure, by whoever is available.
The strongest argument for it is the one nobody makes in the design review: it is what a provider migration costs. The market moves. Models are retired on published calendars — OpenAI keeps a running list of deprecations, and it is longer every year — prices change, and a capability that is worth having in 2026 will not be served by the same endpoint in 2029. An estate behind a gateway changes a routing rule. An estate without one changes application code in every repository that ever called a model.
Why the record cannot be added later
The record is the part that answers questions about the past, and questions about the past are the only kind anyone asks when something has gone wrong.
It has to carry the query, the retrieved material and its versions, the assembled context after truncation and re-ranking, the model and prompt versions that produced the answer, the answer itself, and what the human on the other end did with it. Persisted with the output, not in an application log with a rotation policy.
The reason it cannot be retrofitted is arithmetic rather than engineering. If the index has been re-embedded twice, the source documents are stored as “latest”, and the trace was a debugging convenience that was never written down, then the system cannot be interrogated — only re-run, against a corpus that no longer exists. That is not the same thing and it will not satisfy anybody who is asking.
The part that is genuinely a judgement call
Layer four — whether to serve on somebody else’s endpoint or your own hardware — is the decision that gets argued hardest and matters least at the start. Hosted serving is the right default for almost every organisation until one of three things is true: the data cannot leave a boundary, the volume is high and stable enough that utilisation can be kept up, or latency requirements are tight enough that a network hop to a third party is the problem.
That last set of conditions is not rare, and it is becoming less rare in Europe specifically, where residency and operational-control requirements are moving from preference to procurement condition. But it is a decision that a gateway makes reversible and a direct integration makes structural, which is the actual argument for building the boring layer first.
What this does not tell you
This is a shape, not a product list. It says nothing about which gateway, which store or which serving stack — those depend on the estate, the skills on the ground and what the organisation already runs, and a reference architecture that names vendors is a procurement document wearing a diagram’s clothes.
It also does not claim that every system needs all five layers on day one. A single team with one use case and no regulatory exposure can reasonably start with three. What it should not do is start with three and never write down which two are missing, because that omission is the thing that becomes a rebuild.
The reader who decides differently here is whoever owns the second use case. The first one can be built any way at all. The second is where an organisation finds out whether it built a system or a demonstration — and by then, the cost of the answer is already fixed.