Validation accuracy is high; live predictions look random. Feature leakage: training used information not knowable then—post-inspection results, month-end inventory, statistics computed with future labels. Leakage cheats in backtest; production has no cheat sheet, so performance cliffs. Projects must gate on time splits and feature review—no go-live without passing.
Backtests you cannot reproduce with then-available data are not production-ready.
Only Use What Was Visible Then
Split train/validation by business time—no random shuffle on time-series samples. Feature dictionary notes availability timestamp. Derived stats use past windows only. Labels and features from separate pipelines—avoid one wide table silently importing the future.
- Before launch, replay with "as-of" snapshots—drop below threshold means leakage.
- Business experts review features for hindsight bias.
- Monitor production; cliffs mean check leakage before retuning models.
Review Beats Score Chasing
The XYN digital intelligence system treats predictions as advice but still requires explainable feature timestamps. Models that pass leakage review belong in process; score-chasers lose trust in week one. After trust is gone, honest models stay unused.
List availability time for every feature. If you cannot write it, drop the feature before retraining.
