Your backtest knows more than your system did
A chronological split does not establish that historical inputs were available at prediction time. Replay arrival, publication and revision boundaries before using a backtest to approve the proposed service.
Keep development and test groups separate before comparing performance. No measured results are shown.
Consider a constructed replenishment example. A Monday sale arrives at the warehouse on Wednesday, after Tuesday’s order. Months later, the corrected history lists the sale under Monday. Selecting all sales dated before Tuesday now gives the candidate model information the ordering system could not use.
Evaluate the information boundary as well as the train/test boundary. Chronological periods and an incumbent comparison can be valid while the feature join still leaks later information. The evaluation needs the order in which the serving system could obtain each input.
The data owner should require evidence that every input was retrievable through the intended path at prediction time. A date column or a “point-in-time” label cannot settle that question without a defined timestamp meaning and version history.
The timestamp needs a meaning
Event time records when something happened. Arrival time records when a source received it. Feature availability records when a model could retrieve the derived value. In the hypothetical example, even Wednesday’s arrival might be followed by a batch job that publishes the updated sales aggregate later. A backtest using warehouse arrival as its cut-off could still get ahead of the serving system.
Point-in-time joins are useful precisely because they select historical feature values relative to a requested timestamp. Databricks documents an AS OF join that returns the latest matching value at or before that timestamp. It also explicitly distinguishes this feature lookup from Delta Lake time travel. A historical table snapshot and a correctly timed feature lookup solve different problems. Databricks documentation
The join cannot establish what its timestamp means. Joining on the date of the
sale answers a different question from joining on the time the sales aggregate
became available. Calling both columns timestamp does not make them equivalent.
Tecton’s training-data documentation makes the prediction cut-off explicit and describes joining against an internally calculated effective timestamp for the feature. For batch features, that calculation uses the scheduled job start and configured data delay. This is a useful implementation detail to inspect when assessing a platform. A schedule-based availability rule still needs checking against the delays and failures in your own pipeline. Tecton documentation
What could Tuesday’s order know?
Swipe or scroll for the full diagram →
Sale recorded: excluded from earlier inputs
The Monday event arrives on Wednesday, after Tuesday’s prediction. Its event date does not make it eligible.
Event: 2026-09-07T09:00:00Z. Available: 2026-09-09T09:00:00Z.
- Prediction boundary
- 2026-09-08T09:00:00Z. Availability after this boundary excludes a value from the earlier input, even if its event date is earlier.
- Sale recorded
- Event 2026-09-07T09:00:00Z; available 2026-09-09T09:00:00Z. The Monday event arrives on Wednesday, after Tuesday’s prediction. Its event date does not make it eligible.
- Aggregate published
- Event 2026-09-09T09:00:00Z; available 2026-09-09T12:00:00Z. Publication follows arrival. The serving system cannot retrieve the aggregate before publication.
- Outcome matures
- Event 2026-09-10T09:00:00Z; available 2026-09-10T12:00:00Z; label mature 2026-09-14T09:00:00Z. A later adjudicated outcome can be an evaluation label; it cannot become an earlier model input.
Constructed dates extend the article’s hypothetical Monday-to-Wednesday example; they are not an observed pipeline.
Event, feature availability, prediction and label maturity are distinct boundaries.
Reviewed 2026-10-02
Backfilling can change the question
A backfill can repair data and apply consistent transformations. It can also create values absent from the historical serving path. State whether the evaluation describes the original service or a proposed improved pipeline, and test the availability assumptions for that pipeline.
Tecton’s troubleshooting documentation gives a concrete instance. For built-in aggregations, retrieval from materialised data can exclude records that arrived after the aggregation window was processed. Retrieval from source can include those late arrivals. The resulting difference reflects which history was read. The documentation identifies materialised retrieval as the route that minimises training and serving skew in this situation. Tecton’s late-arriving data guidance
Corrections create the same problem without a delayed feed. A value can be published promptly and revised later. The Federal Reserve Bank of St. Louis makes this distinction visible in its FRED API: the default real-time period describes what is known today about the past. Changing that period supports ALFRED-style queries for what was known at an earlier date. An observation’s date alone does not identify its historical version. FRED API documentation
For an operational dataset, preserving that distinction means retaining the versions needed for the replay. Overwriting a corrected value while keeping its original event date destroys the evidence of what the earlier decision received. Adding an arrival column now cannot recover the overwritten version.
Our argument for a statistical baseline before a GPU requires a fair comparison on held-out data. This is a separate condition on that comparison: the candidate and baseline need inputs that were eligible at the same decision cut-off. A chronological split determines which cases are held out. It does not establish the historical availability of their features.
Make the replay demonstrate its boundary
Start with one inspectable prediction record. Identify the prediction time, feature values, source versions and availability evidence. Compare reconstructed inputs with production records where available. Resolve discrepancies at that boundary before interpreting the downstream model score.
The dataset acceptance criteria should make the following checks executable:
- Fix the prediction moment. State whether the decision occurs before an order is submitted, after it is accepted, or at another named event. Use that cut-off for each case. Keep the later outcome used as the label separate from the information permitted as an input.
- Trace availability through the pipeline. Identify the source version, transformation version and publication boundary behind each feature. Record whether availability is observed or inferred from a schedule. Include the stale value or missing-value behaviour the serving path would actually return.
- Exercise late arrivals and revisions. Take a small constructed history, replay a decision, then introduce a delayed event and a subsequent correction. With the replay definition fixed, rerunning the earlier decision should retain the inputs available then. Later decisions should see the changes when they become eligible.
- Explain every reconstruction gap. Identify features whose old values, publication times or transformation versions cannot be recovered. Measure the candidate again without relying on those uncertain inputs, or collect evidence prospectively before using the result to approve deployment.
Keep the constructed cases beside the feature pipeline. A backfill, schema change or new retrieval mode should have to pass them before its output replaces an accepted evaluation dataset. Otherwise a maintenance change can improve the reported score by changing the information available to the test.
A replay that can fail
Does the replay retain only inputs available at the decision?
- Fixture
- One delayed sale and one later correction in a constructed history.
- Procedure
- Freeze the Tuesday prediction boundary.
- Introduce Wednesday arrival and publication.
- Replay Tuesday; check that its eligible input set is unchanged.
- Replay a later decision; check that eligible new information appears.
- Result to check
- Earlier decisions retain their earlier input set.
- Evidence limit
- A constructed replay tests a boundary, not model performance or real pipeline reliability.
- Next test
- Compare reconstructed inputs with production input logs where available.
Method notes derived from the article’s existing replay procedure.
A four-step boundary check keeps later arrivals out of earlier inputs.
Reviewed 2026-10-02
Decide which system the result describes
An exact replay is not always possible, and it is not always the question. A team may be evaluating a proposed pipeline that delivers data sooner than the current one. That is a legitimate design experiment. Its score is conditional on the new arrival and publication assumptions, which need their own operational evidence. It cannot establish how the existing system would have performed.
Correct feature timing also does not establish that labels are sound, that the sample represents future demand, or that the model remains useful when behaviour changes. It resolves a specific source of leakage. A later label is entirely proper when it measures the outcome the model was meant to predict, provided that outcome has not entered the earlier inputs.
Where the historical inputs cannot be established, narrow the performance claim or gather prospective predictions with their actual inputs. If the candidate depends on a new data path, approve that path against its own operating evidence. Do not attribute performance on a repaired, richer history to a service that lacked those inputs.