How to Build a Simple Forecasting Journal
Why keep a forecasting journal
If you want to improve your Brier score, you need feedback. A forecasting journal gives you structured feedback on:
• why you chose a probability
• what evidence you used
• what you missed
• how your confidence level performs over time
Without a journal, most people only remember outcomes, not decision quality.
The minimal journal template
You can journal in a spreadsheet, Notion, or a simple form. The key is consistency.
Fields to log for every forecast
• Question or market (short name)
• Timestamp (when you made the forecast)
• Forecast probability (p)
• Horizon (time to resolution) or your checkpoint rule
• Category (sports, politics, macro, etc.)
• Base rate (your prior)
• Key evidence (3 bullets max)
• What would change my mind (one trigger)
• Confidence notes (why not higher or lower)
Start from base rates
The most common journaling mistake is skipping the base rate step.
A simple rule:
• write your prior first (base rate or market consensus)
• then write the reason for moving away from it
This reduces overconfidence and makes your updates explainable.
Use a “one line thesis”
Force yourself to write one sentence:
• “I think YES is 63% because X, while the main risk is Y.”
This prevents vague forecasting and makes postmortems useful.
Postmortems: the 5 minute review
When the event resolves, add two lines:
• What happened (outcome and key driver)
• Was my process good (yes or no, with one reason)
Do not rewrite history. Your goal is to learn where your probabilities were wrong, not to defend them.
Turn journal entries into calibration feedback
Once per week or month, summarize:
• Brier score by probability bucket (0.50-0.60, 0.60-0.70, etc.)
• where you are overconfident vs underconfident
• which categories dominate your errors
This turns journaling into a loop that directly improves calibration.
What to avoid
• long essays per forecast (you will quit)
• vague reasons like “feels likely” without evidence
• changing your stated probability after the outcome
• only journaling wins
Optional add ons that help a lot
• link to sources or screenshots
• record a “market price at entry” if you are benchmarking to market consensus
• tag whether the forecast qualifies for a challenge rule (coverage, checkpoint, etc.)
Takeaway
A simple journal improves forecasting faster than reading more theory. Log the probability, the prior, the evidence, and a trigger that would change your mind. Then review outcomes by bucket to fix calibration and stop repeating the same mistakes.