Your backtest knows more than your system did

Consider a hypothetical replenishment model. A sale happens on Monday. The shop’s feed reaches the central warehouse on Wednesday. The order was placed on Tuesday. When the team evaluates a replacement model months later, the sale sits neatly in Monday’s history. A query for sales before Tuesday includes it. The historical record is accurate. The proposed model has just been given information that the ordering system could not have used.

The team has already split training and test periods chronologically. It has held back the recent period and compared against the incumbent. None of those checks asks when the inputs became usable. A backtest can respect the order of events while violating the order in which the system learnt about them.

For the data owner approving the evaluation dataset, the acceptance question is whether each input was available through the intended serving path at the moment of prediction. That requires more than a date column and a promise that the join is point-in-time correct.

The timestamp needs a meaning

Event time records when something happened. Arrival time records when a source received it. Feature availability records when a model could retrieve the derived value. In the hypothetical example, even Wednesday’s arrival might be followed by a batch job that publishes the updated sales aggregate later. A backtest using warehouse arrival as its cut-off could still get ahead of the serving system.

Point-in-time joins are useful precisely because they select historical feature values relative to a requested timestamp. Databricks documents an AS OF join that returns the latest matching value at or before that timestamp. It also explicitly distinguishes this feature lookup from Delta Lake time travel. A historical table snapshot and a correctly timed feature lookup solve different problems. Databricks documentation

The join cannot establish what its timestamp means. Joining on the date of the sale answers a different question from joining on the time the sales aggregate became available. Calling both columns timestamp does not make them equivalent.

Tecton’s training-data documentation makes the prediction cut-off explicit and describes joining against an internally calculated effective timestamp for the feature. For batch features, that calculation uses the scheduled job start and configured data delay. This is a useful implementation detail to inspect when assessing a platform. A schedule-based availability rule still needs checking against the delays and failures in your own pipeline. Tecton documentation

Historical record	Replay requirement
Sale dated Monday	Keep its event date for the sales window.
Feed received Wednesday	Exclude it from Tuesday’s available inputs.
Aggregate published after arrival	Admit the revised feature only after it becomes retrievable.

Backfilling can change the question

Recomputing features from the source is attractive. It applies consistent code, repairs omissions and produces a cleaner dataset. It can also produce a history that the online system never saw.

Tecton’s troubleshooting documentation gives a concrete instance. For built-in aggregations, retrieval from materialised data can exclude records that arrived after the aggregation window was processed. Retrieval from source can include those late arrivals. The resulting difference reflects which history was read. The documentation identifies materialised retrieval as the route that minimises training and serving skew in this situation. Tecton’s late-arriving data guidance

Corrections create the same problem without a delayed feed. A value can be published promptly and revised later. The Federal Reserve Bank of St. Louis makes this distinction visible in its FRED API: the default real-time period describes what is known today about the past. Changing that period supports ALFRED-style queries for what was known at an earlier date. An observation’s date alone does not identify its historical version. FRED API documentation

For an operational dataset, preserving that distinction means retaining the versions needed for the replay. Overwriting a corrected value while keeping its original event date destroys the evidence of what the earlier decision received. Adding an arrival column now cannot recover the overwritten version.

Our argument for a statistical baseline before a GPU requires a fair comparison on held-out data. This is a separate condition on that comparison: the candidate and baseline need inputs that were eligible at the same decision cut-off. A chronological split determines which cases are held out. It does not establish the historical availability of their features.

Make the replay demonstrate its boundary

Start the review with a decision record. Ask the team to show its prediction time, the feature values used and the evidence that each value was retrievable then. Where production input logs exist, compare the reconstructed inputs against them. A model score is too far downstream to diagnose a discrepancy at this boundary.

The dataset acceptance criteria should make the following checks executable:

• Fix the prediction moment. State whether the decision occurs before an order is submitted, after it is accepted, or at another named event. Use that cut-off for each case. Keep the later outcome used as the label separate from the information permitted as an input.
• Trace availability through the pipeline. Identify the source version, transformation version and publication boundary behind each feature. Record whether availability is observed or inferred from a schedule. Include the stale value or missing-value behaviour the serving path would actually return.
• Exercise late arrivals and revisions. Take a small constructed history, replay a decision, then introduce a delayed event and a subsequent correction. With the replay definition fixed, rerunning the earlier decision should retain the inputs available then. Later decisions should see the changes when they become eligible.
• Explain every reconstruction gap. Identify features whose old values, publication times or transformation versions cannot be recovered. Measure the candidate again without relying on those uncertain inputs, or collect evidence prospectively before using the result to approve deployment.

Keep the constructed cases beside the feature pipeline. A backfill, schema change or new retrieval mode should have to pass them before its output replaces an accepted evaluation dataset. Otherwise a maintenance change can improve the reported score by changing the information available to the test.

Decide which system the result describes

An exact replay is not always possible, and it is not always the question. A team may be evaluating a proposed pipeline that delivers data sooner than the current one. That is a legitimate design experiment. Its score is conditional on the new arrival and publication assumptions, which need their own operational evidence. It cannot establish how the existing system would have performed.

Correct feature timing also does not establish that labels are sound, that the sample represents future demand, or that the model remains useful when behaviour changes. It resolves a specific source of leakage. A later label is entirely proper when it measures the outcome the model was meant to predict, provided that outcome has not entered the earlier inputs.

If the historical boundary cannot be reconstructed, the data owner has a concrete choice: narrow the performance claim or collect predictions and their actual inputs in a prospective shadow run. Approving deployment on a richer, repaired history leaves the production system responsible for a result achieved with information it never had.