Coverage
Coverage is the share of eligible questions you actually forecast. Low coverage can hide selection bias and makes performance comparisons less reliable.
Definition
Coverage measures how much of the available question set you actually forecast. If 1,000 questions were eligible and you forecast 200, your coverage is 20%.
Why it matters
Low coverage makes it easier to look good by avoiding hard questions. That is a form of selection bias. High coverage makes evaluation more representative and improves confidence in your track record.
How to report it
Scorecards should ideally include:
• eligible questions (denominator)
• forecasted questions (numerator)
• coverage percentage
Related
Coverage connects to fairness in leaderboards and to out of sample evaluation. Pair it with Brier score or Brier skill score for a more complete picture.