Re-embedding is a migration, and nobody plans it
Changing the embedding model invalidates every vector in the index, and the index is usually the only copy of how documents were chunked. It is treated as a configuration change, and it is closer to a database migration with no rollback.
Somebody proposes moving to a newer embedding model. The benchmark numbers are better, it costs less per million tokens, and the change is one line in a configuration file. There is no obvious reason not to.
What actually happens is that every vector in the index becomes meaningless, because vectors from two different models do not share a space and comparing them produces a number that is arithmetically valid and semantically nothing. So the corpus has to be re-processed — which means the chunking has to run again, which means somebody has to find the chunking code, which means the question of what the corpus was at the time is suddenly urgent.
The claim: a retrieval index is a derived artefact whose derivation is almost never reproducible, and the first time that matters is the day you want to change the embedding model.
What the index actually depends on
- Input 01 The source documents At the version they were on when they were ingested — not the current version.
- Input 02 The extraction How text came out of the original format. A parser change silently alters every downstream chunk.
- Input 03 The chunking rules Size, overlap, and what counted as a boundary. Usually a constant in a script.
- Input 04 The metadata attached Permissions, dates, source system. What the filters run against.
- Input 05 The embedding model Including the exact version. This is the one people know about.
Only the last of these appears in most system documentation, and it is the least consequential of the five for the quality of retrieval. Chunking decides what can be retrieved at all. Extraction decides what is in the chunks. Metadata decides who can see them. An organisation that has recorded only the embedding model has recorded the part it can look up.
The migration that is not planned as one
- 01 Reprocess Every document, through extraction and chunking, at a known version.
- 02 Re-embed The whole corpus. Cost and elapsed time scale with the corpus, not the change.
- 03 Re-evaluate Against a retrieval evaluation set, or the change is unmeasured.
- 04 Cut over With both indexes live, because rollback means the old vectors still exist.
Stage three is the one that is skipped, and skipping it is why some re-embeddings quietly make a system worse. A newer model with better benchmark scores can retrieve differently on a specific corpus — particularly one with domain vocabulary, tables, or short documents — and nobody finds out, because the only test performed is that the system still returns answers.
A retrieval evaluation set is a modest thing: a few hundred real questions, each with the documents that should have been retrieved, marked by someone who knows the domain. It is not the same as an end-to-end evaluation set, because it measures the retrieval step rather than the answer, and a system can retrieve the wrong material and still produce a plausible answer — which is exactly the failure that a general quality metric hides.
Why this is a governance problem, not only an engineering one
If a system’s answers can be questioned later — and any system supporting a consequential decision can be — then the question is what the index contained at the time of the answer. That has an answer only if the index carries a version that changes when any of the five inputs above changes, and only if the trace recorded which version answered.
Without it, the honest response to a question about a past answer is that the system can be re-run against a corpus that no longer exists. That is not a reconstruction. It is a new experiment presented as an audit.
The cost argument, since it is usually what unlocks the work
Re-embedding an established corpus is not free, and the cost is proportional to the corpus rather than to the size of the change. That is worth knowing before committing to a provider whose embedding models are on the same kind of published retirement schedule as everything else — Anthropic, OpenAI and the rest publish model deprecation calendars and honour them, which means at some point a re-embedding is not a choice.
An organisation that knows this in advance makes two cheap decisions differently. It keeps the extraction and chunking pipeline reproducible from the start, so the reprocessing is a run rather than an archaeology project. And it treats the choice of embedding model as a decision with a switching cost attached, rather than as a default inherited from a tutorial.
What this does not tell you
There is no general answer to which embedding model is right, and this is not an argument for staying on an old one. Retrieval quality is corpus-specific to a degree that surprises people, and the only reliable way to choose is to measure on your own material.
It also does not follow that every index needs full reconstructibility. A prototype does not. What it needs is a note in the design record stating that the index cannot currently be reproduced, so that the decision to industrialise it is made with that knowledge rather than in ignorance of it.
The reader who should act is whoever owns the retrieval system. Ask one question: if we had to rebuild this index exactly as it is today, could we. The answer is usually no, and it is cheap to fix this week and expensive to fix after the corpus has been growing for another year.