Confidence Scores in AI Predictions Explained
A prediction without a confidence estimate is nearly useless. But most confidence numbers are not probabilities, and treating them as such is a mistake.
Two forecasts both say "up". One is produced in a calm, trending market where every model in the ensemble agrees. The other is produced an hour before a central bank decision with models pointing in different directions. Reported identically, they mislead. This is what a confidence score exists to fix.
What the number usually is not
Most confidence scores are not calibrated probabilities. A neural network's softmax output of 0.9 does not mean the event happens 90% of the time — deep networks are systematically overconfident, and financial data makes this worse because the signal-to-noise ratio is so low.
Treating an uncalibrated score as a probability leads directly to over-sizing positions on forecasts that were never that certain.
What it can legitimately represent
- Model agreement. In an ensemble, how closely the components concur. This is one of the more honest confidence proxies available.
- Historical reliability in similar conditions. How accurate this model was previously when volatility, volume and trend looked like they do now.
- Forecast dispersion. The width of the predicted range. A tight distribution genuinely reflects more certainty than a wide one.
- Data quality. Missing data, stale quotes or thin volume should reduce confidence regardless of what the model outputs.
How to read one sensibly
Use it comparatively, not absolutely. A score of 80 does not mean an 80% chance of being right. It means this forecast is on firmer ground than one scoring 55 from the same system. The ordering is informative even when the absolute value is not.
Confidence tells you how much to trust this prediction relative to others from the same model. It does not tell you the odds.
What high confidence still cannot do
No score anticipates genuinely unexpected news. A model can be maximally confident based on everything knowable and be immediately wrong because something unknowable happened. Confidence quantifies uncertainty within the model's view of the world, not the possibility that the world does something outside it.
Practical use
The most defensible use of a confidence score is as a filter rather than a multiplier: ignore forecasts below a threshold instead of scaling exposure up on high ones. Filtering reduces activity in conditions the model handles poorly. Scaling up amplifies exactly the concentration risk that ruins accounts when a confident prediction fails.
Educational content only. Not financial advice. No prediction system can guarantee outcomes.
See the models in action
Reading about forecasting is one thing. Create a free account and get 5 AI predictions a month — no card required.
Start free
