Establish the statistical baseline before you buy a GPU

There is a specific conversation we have had enough times to recognise it from the first sentence. A team has built or bought a large model for a prediction task. It performs well. Somebody senior asks how much better it is than what the organisation was doing before, and the room goes quiet.

Not because the answer is embarrassing. Because nobody measured.

The baseline is not a formality

A baseline is the simplest defensible method applied to the same problem, on the same data, evaluated the same way. For most enterprise prediction tasks that means a regression, a gradient-boosted tree over the obvious features, or — often the most revealing of all — the existing manual process, scored honestly.

Its purpose is not to be beaten. Its purpose is to make the comparison possible at all.

Without one, “the model performs well” is an unanchored statement. Ninety-two per cent accuracy is excellent if the base rate is fifty and unremarkable if the base rate is ninety-one. Teams know this in the abstract and skip it in practice, because the baseline is the least interesting week of the project and the pressure to show something impressive arrives early.

Three things a baseline tells you that nothing else will

Whether the problem is worth a model at all. A meaningful fraction of tasks brought to us as machine learning problems are answered adequately by a rule, a lookup, or a better form. That is not a disappointing outcome. It is the cheapest possible outcome, and it is available only to teams who measured the simple thing first.

Where the ceiling is. If a regression on the obvious features reaches most of the achievable performance, the remaining headroom tells you what a larger model could possibly be worth. Sometimes that headroom does not cover the inference bill, and it is much better to learn this before signing for capacity.

Whether your data is the problem. A baseline that performs poorly and a large model that also performs poorly is a strong signal that the constraint is upstream. That is a data engineering finding wearing a modelling costume, and no amount of model selection will fix it.

The procurement version of this argument

The commercial case is blunter than the scientific one. A vendor demonstration is performed on the vendor’s evaluation. A baseline is performed on yours.

Any supplier proposing a model for your problem should be asked, before contract, what their approach achieves relative to a simple method on your data, measured on a set you control. That question is not adversarial. A good supplier will have asked it themselves, and will be pleased to have a number to point at.

A supplier who cannot answer it — or who explains why a comparison would not be meaningful — has told you something important about how the engagement will go.

What “properly” means here

Baselines get skipped partly because they are boring and partly because doing one badly is worse than not doing one at all. A few things make the difference:

• Agree the metric before you see any results. The single most common way a baseline becomes useless is a metric chosen after the fact to make the preferred method win.
• Split the data the way the system will actually be used. If predictions are made forward in time, the split must be forward in time. A random split on temporal data produces a number that will not survive contact with production.
• Score the incumbent process too. It is the real thing being replaced, and it is usually better than anyone remembers.
• Publish the uncertainty. A point estimate with no interval invites a comparison it cannot support. Two methods separated by less than the noise are not separated.
• Keep the set. The evaluation set is the asset. Every subsequent model change is measured against it, and it is the only reason anyone will believe the fourth version is better than the third.

The uncomfortable case

Occasionally the baseline wins, or comes close enough that the larger model cannot justify its cost. This is the outcome teams quietly hope to avoid, and it is the outcome with the highest return.

We have written a recommendation against adopting an approach that a client was enthusiastic about, on the basis of a study that took five weeks. That is an uncomfortable document to hand over. It is also the single clearest demonstration of what an independent architect is for: we do not bid for the build we specify, so a recommendation not to build costs us nothing to write and saves the client the whole programme.

If the honest answer is that a regression already solves your problem, that is the answer you should get. The expensive part of applied machine learning has never been the model. It has been finding out, eighteen months later, that you did not need it.