Your data records a process, not the world
Historical enterprise data is a record of what an organisation decided, who it decided about, and what it happened to write down. A model trained on it learns the process — including the parts nobody would defend if they were written as a rule.
The dataset is described as ten years of decisions and it is presented as a strength. Ten years is a long time, the volume is large, and the labels are real outcomes rather than anyone’s opinion. It is genuinely a better starting point than most teams have.
What is rarely said out loud is what the labels are labels of. They are not records of what was true. They are records of what the organisation concluded, about the cases it happened to see, using the information it happened to capture, under the policies that were in force at the time. A model fitted to them learns that process. It cannot learn anything else, because nothing else is in the data.
The claim: historical operational data encodes the decision process that produced it, and the failure modes that follow are structural rather than statistical.
The three ways the record differs from the world
| What the column contains | What it is read as |
|---|---|
| The outcome for cases that were approved | The outcome for all cases, including the ones that were declined |
| What an assessor recorded, in the fields available | What was true about the case |
| Decisions under the policy in force that year | A stable relationship between features and outcome |
| Cases that reached the organisation at all | The population the system will be applied to |
The first row is the one with the sharpest consequence and it has a name: selective labels. If a lending, hiring or claims process only observes outcomes for the cases it accepted, then the training data contains no evidence about the cases it rejected — and a model fitted to it will reproduce the historical acceptance boundary while appearing to predict the outcome. It is not measuring risk. It is measuring the previous system’s opinion of risk, and its errors are invisible precisely where the old system was wrong.
The fourth row is the one that quietly breaks deployments. A model trained on cases that arrived through one channel is applied to cases arriving through three, and its performance degrades in a way that looks like drift and is actually a change of population.
The feedback loop that closes afterwards
- 01 Model scores a case Trained on historical decisions.
- 02 The score shapes the decision Directly, or through what a reviewer looks at first.
- 03 The decision is recorded As an outcome, indistinguishable from an unassisted one.
- 04 The next model trains on it Now fitting to its predecessor, not to the world.
This closes within a year in most operational settings, and once it has closed, performance metrics stop measuring what anybody thinks they measure. Agreement between the model and the recorded outcome goes up, because the model influenced the outcome. Nothing in a standard evaluation distinguishes that from getting better.
The remedy is not complicated and it is almost never implemented: keep a portion of cases decided without the model’s involvement, and record which cases those were. It costs something. It is the only source of information about what the system would have got wrong, and without it an organisation loses the ability to answer whether the model is helping — permanently, not temporarily.
What to establish before training on operational history
The second question routinely reveals that a dataset spans two or three different processes wearing the same schema. That is not a reason to discard it. It is a reason to treat the policy period as a feature, and to be sceptical of any relationship that does not survive being tested within a single period.
The third question is the one that finds the fields nobody should model on. Optional free-text fields completed by assessors under time pressure are systematically completed more often for unusual cases, which makes their presence a proxy for something the organisation would never knowingly use as one.
Why this belongs in a governance conversation
Every item above is also an equality question. A model that reproduces a historical acceptance boundary reproduces whatever was uneven about it, and the usual mitigations — removing protected characteristics from the feature set — address the least of the mechanisms. The characteristic is not in the column. It is in the process that generated the labels, and it survives the column being deleted.
This is why the useful artefact is not a fairness metric computed at the end. It is a written account of how the data came to exist, produced before the modelling starts, saying what was observed, what was not, and what changed. That document is what makes an impact assessment answerable, and it is the part of the work that regulators, auditors and courts will find most legible — the UK Information Commissioner’s guidance on AI and data protection puts the same emphasis on being able to explain the provenance of what a system learned from.
What this does not tell you
None of this says the data is unusable. Most of it is usable, and the alternatives — synthetic data, a smaller purpose-collected set, no model at all — carry their own problems, some of them worse. The argument is that the limitations should be written down before the model is built, because afterwards they become criticisms of somebody’s work rather than properties of the material.
It also does not offer a statistical fix. Techniques exist for selective labels and for feedback loops, and they help, and none of them recovers information that was never recorded. The honest position is that some questions cannot be answered from the data available, and saying so is more valuable than a model that answers them anyway.
The person who should act is whoever owns the dataset. Write the provenance account first — two pages, before the modelling starts. It is the cheapest document in the project and it is the one that will still be read in three years.