Miscalibration
Miscalibration is the gap between predicted probabilities and observed frequencies. It is the core failure mode behind unreliable probabilities.
Definition
Miscalibration means your probabilities do not match reality in aggregate. If you say 70% repeatedly and the event happens only 55% of the time, you are miscalibrated in that range.
How to detect it
• Use a reliability diagram or calibration curve.
• Review per probability bucket realized frequencies.
Two common patterns
• Overconfidence: probabilities are too extreme.
• Underconfidence: probabilities are too close to 50%.
Why it matters
Miscalibration increases error and worsens Brier score. Improving calibration often produces the fastest gains in scorecards, especially if you frequently publish high confidence forecasts.