A confidence threshold belongs to the model version

A model reaches its retirement date. The replacement performs slightly better on the evaluation set, the integration test passes, and the team changes the model identifier. One line of configuration remains untouched: send every case scored below 0.8 to a person.

The threshold looks like business policy. It was agreed with the operation, written into a control and used to size the review queue. Yet 0.8 is only a position on one model’s score scale. A replacement model can assign different scores to the same cases while making the same number of correct predictions.

A confidence threshold belongs to the model version that produced the confidence. A migration is incomplete until the new model’s scores have been related to observed outcomes and a new operating point has been approved. Keep the decision rule if it still serves the process. Do not keep the old number by default.

Accuracy does not preserve the score scale

Suppose a hypothetical classifier routes low-confidence cases to human review. On the old model, a score of 0.8 might identify a band in which about eight of every ten predictions are correct. That interpretation holds only if the model is calibrated for the population in question. It means that among predictions given 0.8 confidence, the expected proportion correct is 0.8.

Guo and colleagues use that definition in their study of neural-network calibration. They found that modern neural networks could improve classification accuracy while producing confidence estimates that overstated correctness. The architecture, training choices and regularisation affected calibration. Their practical result was that temperature scaling, fitted on a held-out validation set, often improved calibration without changing the predicted class. On Calibration of Modern Neural Networks, ICML 2017

That separation matters during replacement. Accuracy asks how often the chosen class is right. Calibration asks what a score means. Two models can have similar accuracy and different mappings from score to outcome. A model can also become more accurate and less calibrated. Passing an accuracy gate therefore says nothing about whether the old threshold still selects the intended cases.

The decision to preserve	The evidence to rebuild
Which cases may proceed without review	The new score bands and their observed error rates
How many cases the review team can absorb	The new version's score distribution on representative traffic
The relative cost of a miss and an unnecessary review	The new operating curve across plausible thresholds
Which groups require separate protection	Calibration and error measures for those groups where the data supports them

The left column is policy. The right column is measurement. Copying the numeric threshold treats them as one object and lets a technical migration silently change the policy’s effect.

Recalibration is part of migration acceptance

Start with scores from the frozen replacement candidate on data that resembles the work it will receive. Do not fit calibration on the training examples. Use held-out, labelled cases and retain enough untouched data to test the result. Plot a reliability diagram or an equivalent table that groups predictions by score and compares confidence with observed correctness. Report how many cases sit in each band, because a visually neat band supported by few observations cannot carry a consequential decision.

Then evaluate operating points. For each plausible threshold, show the errors that would pass, the correct cases that would be escalated, and the volume sent to the human queue. Where consequences differ by group, check those quantities separately rather than allowing the largest group to determine one reassuring average. The threshold owner can then choose the point that implements the existing risk and capacity decision, or decide explicitly that the decision should change.

This record is more useful than a sentence saying that the successor passed the same benchmark. It establishes what the number controlling the workflow means under the replacement. It also gives the production monitor a baseline for noticing when that meaning moves.

A calibrated threshold can still expire

Calibration on a held-out set is conditional on the population represented by that set. Guo and colleagues state that their calibration methods assume the training, validation and test data come from the same distribution. Production rarely grants that assumption indefinitely. On Calibration of Modern Neural Networks, paper

Ovadia and colleagues tested uncertainty methods under increasing dataset shift across image, text, categorical and genomics settings. In their experiments, accuracy and the quality of uncertainty generally degraded as shift increased. Temperature scaling fitted on independent and identically distributed validation data worked under that condition and small shifts, then lost ground as shift grew. The study does not provide a universal expiry period. It shows why a calibration result from migration day cannot be treated as permanent. Can You Trust Your Model’s Uncertainty?, NeurIPS 2019

The production design therefore needs current labels, not just score and input drift. Monitor the relationship between score bands and outcomes on the traffic the system actually receives. A changed score distribution can tell the team where to look. Only observed outcomes can show whether the confidence attached to those scores still deserves its interpretation. This is the migration consequence of the institute’s earlier distinction between a drift alert and a performance alarm.

The NIST AI Risk Management Framework places this inside a continuing measure function. It calls for pre-deployment and regular operational testing, measures of uncertainty and documented results, with responses that can include recalibration, mitigation or removal. The framework is voluntary and does not prescribe a calibration method or threshold. It does support the operating principle that uncertainty evidence is maintained across the lifecycle rather than accepted once. NIST AI RMF Core, Measure

What this does not tell you

Calibration does not make a model safe, fair or suitable for a particular use. A well-calibrated model can still have unacceptable error rates, omit a subgroup or optimise the wrong target. A single aggregate calibration measure can also hide serious local errors. The method must be chosen for the decision and the available data.

Nor is temperature scaling a universal remedy. The cited results cover specific models and datasets, and the evidence under shift shows its limit. Some systems produce scores that are not intended to represent probabilities at all. In that case, the team should call them scores, test how they rank cases and avoid presenting them as confidence until an empirical mapping supports that word.

The earlier article on model retirement argued for a runnable evaluation set before the supplier’s deadline arrives. The model owner now has one more acceptance question for that migration. At the threshold the replacement will use, which cases proceed, which reach a person, and what observed evidence says that boundary still implements the decision the organisation approved?