The Base Rate Trap: Why Priors Beat Vibes
Base rates are the starting line
A base rate is the historical frequency of an outcome in a relevant reference class.
Examples:
• how often underdogs win in a league
• how often a bill passes after leaving committee
• how often a startup raises a follow on round within 18 months
Good forecasting usually starts with a base rate, then updates with evidence.
The base rate trap
The base rate trap is when you jump straight to a story and assign a probability without anchoring on priors.
Humans do this because stories feel specific and persuasive, while base rates feel boring.
But base rates are often the single best baseline you have, especially early in the forecast horizon.
Why people neglect base rates
Common drivers:
• vivid recent news (availability bias)
• overconfidence in a narrative
• emotional attachment to a team or outcome
• confusing “possible” with “likely”
Neglecting base rates tends to produce overconfidence and unstable calibration.
A simple base rate workflow
Use a three step workflow for each forecast:
Step 1: write the reference class
Define what “similar” means. This is the hardest part.
Example: “late season match between top 3 and bottom 3 teams” is a better class than “football matches”.
Step 2: write the base rate
Even a rough base rate is better than none.
Write it explicitly in your journal or UI as “prior”.
Step 3: justify your update away from the prior
Now apply evidence. The key question:
• “What new information moves me away from the prior, and by how much?”
If you cannot answer, you should not move much.
Base rates make Brier skill score meaningful
Raw Brier score is hard to compare across datasets.
Using base rate as a benchmark makes Brier skill score meaningful and hard to game.
It also makes leaderboards more fair when markets are thin or market prices are unavailable.
Common ways base rates go wrong
Wrong reference class
A base rate from the wrong class can be worse than no base rate. If your class is too broad, you will miss important structure.
Outdated rates
Worlds change. If your base rate drifts, you need a rolling window or periodic refresh.
Hiding uncertainty
Base rates are estimates too. If the history is small, the base rate itself is uncertain. Treat it as an anchor, not a law.
Practical habit: write priors first
If you want one habit that improves forecasting fast:
• write the prior first
• then write the update reason
This is also the best habit for a forecasting journal.
Takeaway
Base rates are boring because they are powerful. They prevent narrative overreach, reduce overconfidence, and create stable benchmarks for skill scoring. Anchor on priors, then update with evidence, and your calibration will improve.