Why Machine Learning Models Fail in Live Markets
The gap between backtest and live performance has predictable causes. Most of them are known before deployment and ignored anyway.
Almost every model degrades when it meets live markets. The question is whether the degradation is a modest, expected haircut or a total collapse. The difference comes down to a handful of causes.
1. The market changed
Models learn relationships that held during a specific period. Interest rate environments shift, correlations break, volatility regimes turn over. A model trained through a long low-volatility bull phase has genuinely never seen the conditions it may be asked to trade.
This is not a flaw to be engineered away. It is the nature of a non-stationary system, and it argues for periodic retraining and for monitoring live accuracy against expectation.
2. Costs were underestimated
A strategy earning eight basis points per trade before costs and paying ten in spread, slippage and commission is a losing strategy that looks profitable on paper. Backtests that assume execution at the closing price systematically flatter high-frequency approaches.
3. It was overfitted
If you tested a hundred variants and shipped the best one, you selected for a model that fit the noise in your test period especially well. The more configurations you try against the same data, the more the winner's performance reflects luck.
4. Something leaked
Scaling before splitting, forward-looking indicators, restated data. Leakage produces backtests that are not merely optimistic but meaningless, because they measure the model's access to the future rather than its skill.
5. The model affects the market
Relevant at size rather than for individuals, but real: a strategy that moves the price it is trying to capture eats its own edge. Illiquid instruments make this bite far sooner than people expect.
Expect live results to be worse than backtest results. A system that is only viable if the backtest is accurate is not viable.
What reasonable practice looks like
- Walk-forward validation rather than a single split.
- Realistic cost assumptions, erring on the pessimistic side.
- A held-out period never used for any tuning decision.
- Paper trading before capital, to surface execution issues.
- Live monitoring, with a defined threshold at which you stop and re-examine.
None of this guarantees profitability. It does mean that when performance decays, you find out from your monitoring rather than from your account balance.
Educational content only. Not financial advice. Trading involves substantial risk of loss.
See the models in action
Reading about forecasting is one thing. Create a free account and get 5 AI predictions a month — no card required.
Start free
