Brier Score
Brier score measures the accuracy of probabilistic forecasts for binary events by averaging the squared error between predicted probability and the actual outcome. Lower is better.
Definition
Brier score is a scoring rule for probabilistic forecasts of binary events (yes/no outcomes). It measures how close your predicted probability was to what actually happened.
Formula
For one forecast with predicted probability p and outcome o (where o is 1 if the event happened, otherwise 0):
(p - o)^2
Across N forecasts, Brier score is the average:
BS = (1/N) * sum((p_i - o_i)^2)
How to interpret it
Lower is better. A perfect forecaster gets 0. Scores get worse as forecasts move away from outcomes.
Examples:
• You predict p = 0.70 and the event happens (o = 1) -> error (0.70 - 1)^2 = 0.09.
• You predict p = 0.70 and the event does not happen (o = 0) -> error (0.70 - 0)^2 = 0.49.
Why it matters in prediction markets
Brier score rewards honest probabilities and penalizes overconfident mistakes more than mild ones. That makes it useful for measuring forecasting skill and for comparing strategies over time.
Common pitfalls
Confusing accuracy with calibration: You can be right often but poorly calibrated. Use calibration checks alongside Brier score.
Ignoring base rates: If events are mostly yes (or mostly no), simple baselines can look strong. Compare against a benchmark or use Brier skill score.
Selection bias: Scoring only the questions you feel confident about can inflate results. Track selection bias and evaluate out of sample when possible.
Related
If you want a normalized measure versus a baseline, use Brier skill score. For an alternative loss function, see log loss.