Benchmark
A benchmark is a reference forecast used for comparison, such as 50/50, base rate, or market consensus. It is required to compute Brier skill score.
Definition
A benchmark is a reference forecast used to evaluate skill. Instead of asking “is my score good?”, you ask “am I better than this baseline?”
Common benchmarks
• 50/50 for every binary event
• Base rate (“climatology”) using the empirical base rate
• Market consensus from market prices
Why it matters
Benchmarks make comparisons fairer across datasets. They also power Brier skill score, which reports performance relative to the baseline.
Common pitfalls
Picking a weak benchmark: Choosing a naive baseline can exaggerate skill. Prefer base rate or market consensus when the goal is serious evaluation.
Mismatch in timing: A benchmark must be defined at the same time as the forecast (same forecast horizon), otherwise comparisons are distorted.
Related
Benchmarks are used in Brier skill score and for performance reporting in scorecards.