Proper Scoring Rule
A proper scoring rule rewards probabilistic forecasts in a way that makes reporting your true beliefs the best strategy. It discourages hedging or “gaming” the score.
Definition
A proper scoring rule is a rule for scoring probability forecasts where your expected score is maximized (or expected loss minimized) when you report your true probability belief.
Why it matters
Proper scoring rules are the foundation of honest forecasting systems. If a scoring rule is proper, forecasters cannot improve their expected score by systematically misreporting probabilities (for example, always saying 50% to “play it safe”).
Examples
• Brier score is a proper scoring rule for binary events.
• Log loss is also proper and penalizes extreme overconfidence even more strongly.
Common misunderstanding
Proper does not mean perfect: A rule can be proper and still be noisy on small samples. Use enough questions and evaluate out of sample when possible.
Related
Proper scoring rules connect directly to calibration and the incentives in leaderboards. If you are building a contest, using a proper scoring rule helps keep the game fair.