Coverage Bias
Coverage bias is the distortion that occurs when evaluation reflects only the subset of questions a forecaster chose to answer, rather than the full eligible set.
Definition
Coverage bias occurs when measured performance is skewed because you do not forecast a representative set of questions. It is a specific manifestation of selection bias, driven by low coverage.
Why it matters
Low coverage makes it easier to avoid hard questions. That can inflate your score and make comparisons unfair.
How to reduce it
• Define eligibility rules and score all eligible questions.
• Report coverage and participation metrics alongside scores.
• Use leaderboards that reward consistency, not just raw score.
Related
Coverage bias is closely connected to coverage, participation rate, and out of sample evaluation.