What the EU AI Act actually asks of a retrieval system

A retrieval-augmented system is the most common thing being built in enterprises right now, and it is the thing most often assessed last. The pattern is familiar: a vector index over internal documents, a model that reads the retrieved passages, and an answer returned to a person who then does something consequential with it.

When the compliance conversation finally arrives, it almost always starts in the wrong place. Teams open the regulation looking for rules about models — training data, bias testing, accuracy thresholds — and are relieved to find that most of the heavy obligations attach to a classification their system probably does not have.

That relief is misplaced, and not because the classification is wrong. It is misplaced because the obligations that do apply are architectural, they are cheap to satisfy before the build, and they are close to impossible to satisfy afterwards.

The question the Act is really asking

Strip the drafting back and a great deal of the risk-management, logging and human-oversight material resolves into one demand: when this system produced an output that mattered, can you show what it was working from?

Not “what does the model do in general”. Not “what is your average accuracy”. What did this system, on this date, retrieve and act on, for this answer.

For a retrieval system that decomposes into a short and unforgiving list:

• Which documents were in the index at the time.
• Which version of each of those documents.
• Which passages were returned for the query, and in what order.
• What the model was given as context, after truncation and re-ranking.
• Which model version produced the answer.
• What the person on the other end did with it.

Every one of those is a design decision made in the first week of the build. None of them is a control you can add in the last week.

Why this is architecture, not documentation

Consider the most ordinary failure. A customer receives an answer that turns out to be wrong. Six months later somebody asks why.

If the index has been re-embedded twice since, if the source documents are stored as “latest” with no version history, and if the retrieval trace was never persisted because it was only ever a debugging convenience, then the honest answer is that nobody knows. The system cannot be interrogated. It can only be re-run, on a corpus that no longer exists, which is not the same thing and will not satisfy anyone asking the question.

That is not a logging gap. Logging is what you add when you know what you need to record. This is a system that was designed without a memory, and retrofitting one means reprocessing history you did not keep.

If you cannot reconstruct a single answer six months later, you do not have a logging problem. You have an architecture problem that has been rebranded as a logging problem.

The distinction matters commercially, because the two have very different price tags. A retrieval trace persisted alongside an index version is a schema decision and a modest amount of storage. Reconstructing provenance for a system that has been running blind for a year is a project.

What to decide before the first sprint

These are the decisions we look for in an architecture review, and the ones that are missing most often. They are not a compliance checklist — they are the things that make a compliance checklist answerable later.

Version the index, not just the documents. An embedding model change silently invalidates every stored vector. If the index does not carry a version that changes when the embedding model changes, “which index answered this” has no answer.

Persist the retrieval trace with the answer. Document IDs, versions, scores, and the final assembled context. Store it with the output, not in an application log that rotates after thirty days.

Record what was cut. Truncation and re-ranking decide what the model actually saw. The passages you dropped are part of the explanation for the answer you got.

Pin the model version in the record. “GPT-class model” is not a version. A provider’s silent upgrade is a change to your system’s behaviour that you did not make and cannot otherwise detect.

Capture the human decision. The Act’s interest in human oversight is not satisfied by a person being nominally in the loop. It is satisfied by evidence of what that person did — accepted, edited, escalated, overrode — which is also the only data that will ever tell you whether the oversight is real.

Keep the evaluation set under version control. It is the only artefact that lets you say a change made things better rather than merely different, and it is the first thing that rots when nobody owns it.

Classification still matters — it just is not the first question

None of the above depends on where your system lands under the Act. That is deliberate. The classification determines how much obligation attaches; the architecture determines whether you are able to meet any of it.

Classification is also genuinely difficult, and it is the part where the honest answer is usually “it depends on the deployment, not the technology”. The same retrieval system answering internal policy questions and screening job applicants are not the same system in the eyes of the regulation, and the difference is in the use, not the code.

The consequences of getting it wrong are not small. The Act’s Article 99 sets administrative fine ceilings of up to €35,000,000 or 7% of total worldwide annual turnover, whichever is higher, for the prohibited-practice breaches, and up to €15,000,000 or 3% for most other obligations. Those are ceilings, not expected outcomes, and they are the only external figures in this piece for a reason: everything else you read about the cost of non-alignment is modelled, and we would rather cite the regulation than somebody’s model of it.

What we are not saying

We are not saying that following the list above makes a system compliant. We do not use that word, and neither should any supplier talking to you about this. Formal interpretation of the regulation as it applies to your organisation is a matter for your legal counsel, and anyone offering you certainty here is selling something they cannot deliver.

What we are saying is narrower and more useful: the difference between a system that can be assessed and one that cannot is decided by six or seven design choices, all of them cheap at the start, and all of them made before anybody in your organisation has thought about the regulation at all.

That is the argument for doing the architecture properly first. Not because a regulator is coming, but because the alternative is a system nobody can explain — and that becomes your problem long before it becomes theirs.