Selection Bias
Selection bias occurs when the set of questions you score is not representative, such as only forecasting “easy” questions or only counting the ones you feel confident about.
Definition
Selection bias happens when performance is measured on a non representative subset of events. In forecasting, this often means you choose which questions to answer based on confidence, interest, or expected success.
Why it matters
Selection bias can make a forecaster look better than they are. If you only forecast “sure things”, your Brier score may improve, but the evaluation no longer reflects real skill on the full question set.
Common patterns
• Only predicting when you already agree with market consensus
• Avoiding long horizon or ambiguous questions
• Excluding bad forecasts from your dataset (“cherry picking”)
How to reduce it
• Pre commit to a list of questions to forecast.
• Score all eligible questions over a fixed time window.
• Compare against a stable benchmark and track coverage (how many questions you actually forecast).
Related
Selection bias is closely linked to evaluation practices like out of sample testing and to fairness in leaderboards.