In a screening system, the false positive is the product
A classifier with excellent accuracy on a rare event still hands its operators far more wrong answers than right ones. That is not a modelling failure, it is arithmetic — and it determines the staffing plan, not just the evaluation report.
The model is presented and it is good. Sensitivity is high, specificity is high, the confusion matrix is on the slide, and the room is satisfied. Somebody asks what happens next and the answer is that flagged cases go to the review team.
Nobody in the room has multiplied the numbers by the volume, and the review team is not represented. Both of those omissions are about to become the project’s main problem.
The claim is old, well documented and almost never applied at the point of design: when the thing you are looking for is rare, the alerts a system produces are mostly wrong even when the system is excellent. The arithmetic is not a criticism of the model. It is a fact about the population, and it makes the review capacity a first-order design constraint rather than a downstream operational detail.
The arithmetic, once
Take a hundred thousand cases a month, and suppose one in a thousand is the thing you are looking for. That is a hundred genuine cases. Now take a model that catches ninety-nine of every hundred genuine cases, and that wrongly flags one in every hundred ordinary ones — figures most teams would be pleased to report.
Roughly eleven hundred alerts, of which about one in eleven is real. The model did what was asked. The reviewers are now looking at ten wrong answers for every right one, and the operational reality is that they will begin to treat the alert as noise — which is the mechanism by which an accurate model produces a worse outcome than no model at all.
This is the base rate fallacy, and it has been documented in intrusion detection for a quarter of a century. The review of the problem in cybersecurity sets out how consistently it is rediscovered, discipline by discipline, and how reliably the discussion stays inside the modelling team instead of reaching the people who have to work the queue.
Why this is a design decision, not a tuning exercise
The instinct is to raise the threshold until the alert volume is bearable. That works, and what it buys is bought from the other column.
| The move | What it spends |
|---|---|
| Raise the threshold | Missed genuine cases. In a screening system, this is the harm the system exists to prevent. |
| Add reviewers | Cost that scales with volume, and a queue whose quality falls as it lengthens. |
| Narrow the population screened | Raises prevalence and fixes the arithmetic — but the narrowing itself is a decision that can discriminate. |
| Stage the screening | Engineering effort now, in exchange for the only version of this that scales. |
The fourth row is where good systems end up. A cheap, high-recall first stage reduces the population, and a more expensive second stage runs only on what survives it. Prevalence in the second stage is far higher than in the first, so its alerts are far more often right, and the total review burden falls without lowering recall. It is more work to build and it is the only one of the four that does not trade the system’s purpose against its running cost.
The third row is where governance has to be present. Screening a subset raises precision by choosing who gets screened, and the choice of subset is exactly the kind of decision that produces disparate outcomes across groups. It is a legitimate design move and it is not a technical one.
What to specify before the model is built
The last two are the ones that make this a governance artefact rather than a metrics table. A false positive is not a rounding error to the person it lands on: it is an account frozen, an application delayed, a claim investigated, a name on a list. Systems that record only the aggregate error rate have no way to see that cost, and no way to notice when it falls unevenly.
And the threshold has to have an owner. In most estates it is changed by whoever is on call when the queue backs up, which means the operating point of a consequential system is set by capacity pressure rather than by anybody’s decision about the balance between missing cases and flagging innocents.
What this does not tell you
The arithmetic above is a worked example with round numbers, not a claim about any real deployment. Prevalence varies enormously by domain, and the whole point of the exercise is that you have to compute it for your population rather than borrowing a figure from someone else’s.
It also does not say that screening systems should not be built. Most of them should. It says that a screening system is a socio-technical system whose capacity, thresholds and review quality are part of the design, and that a project which evaluates only the classifier has evaluated the smallest component of the thing it is deploying.
The person who should read the confusion matrix differently is the one who signs off the operating model. Ask for alerts per reviewer per day before you ask for accuracy. If nobody can produce that number, the system has not been designed — only trained.