Question Difficulty
Question difficulty reflects how hard an event is to forecast given available information. It affects score interpretation and fair comparisons across datasets.
Definition
Question difficulty describes how predictable an event is. Some questions are close to coin flips with little signal, while others have strong leading indicators.
Why it matters
Scores depend on the question set. A forecaster can look strong by focusing on easy questions, which creates selection bias. Difficulty also affects fair leaderboard comparisons across categories.
Practical approaches
• Compare performance to a benchmark (base rate or market consensus) instead of using raw scores alone.
• Segment by category and forecast horizon.
• Report coverage and mix of question types.
Related
Difficulty connects to out of sample evaluation and to how you define a benchmark.