Forecast Error
Forecast error is the difference between your predicted probability and the actual outcome. Scoring rules summarize forecast error across many events.
Definition
Forecast error is the gap between a predicted probability and the realized outcome for a binary event.
One forecast
If your forecast is p and the outcome is o (0 or 1), the raw error is p - o. Scoring rules use transformations of this error to create a stable metric.
How scorecards use it
• Brier score uses squared error: (p - o)^2.
• Log loss uses logarithmic penalties.
Related
Forecast error is influenced by calibration and by how extreme your probabilities are (sharpness).