How We Backtest Our Models

Any forecasting product can publish an accuracy figure. Whether it means anything depends entirely on how it was produced.

Walk-forward, not random splits

Standard machine learning practice shuffles data before splitting. Applied to a price series this is invalid: shuffling means training on later data to predict earlier data, and adjacent observations are so correlated that near-identical rows land in both sets.

Walk-forward validation instead trains on one window, tests on the period immediately following, then rolls forward. Every test period is strictly later than the data that produced the model being evaluated.

Preventing leakage

Leakage is information from the future reaching the model. The most common source is scaling: normalising the whole dataset before splitting means training data carries statistics computed partly from the test period.

The rule that resolves most cases is simple and unforgiving: every transformation must be computable using only data available at that moment in time. That includes scalers, which must be fitted on the training window alone and refitted at each step.

The gap

A small gap is left between the end of training and the start of testing. Because rolling features are computed over a lookback window, the earliest test observations would otherwise be partly derived from training-period data.

What gets measured

• Directional accuracy across multiple periods, not one aggregate
• Performance broken down by instrument and volatility regime
• Consistency — whether results are stable or driven by a few exceptional stretches
• Behaviour during the worst periods, not just the average

The limitations, stated plainly

A backtest is not evidence that a model works. It is evidence that it did not obviously fail on the history available.

Backtests cannot capture future regime changes. They cannot fully model execution reality. And any process involving repeated evaluation carries some risk of fitting to the evaluation data, however carefully separated.

This is why live accuracy is tracked separately and published, and why expecting live results to be somewhat worse than backtested results is the correct default rather than a disappointment.

TradePredictor provides analytical tools for informational purposes only and does not provide financial advice. Backtested performance does not guarantee future results.