Confidence Interval
A confidence interval is a range that reflects uncertainty in an estimated metric, such as the realized frequency in a calibration bucket. Wider intervals usually mean smaller samples.
Definition
A confidence interval is a range that reflects uncertainty around an estimate. In forecasting, it is most often used for:
• the realized frequency inside a probability bucket
• points on a calibration curve or reliability diagram
Why it matters
Calibration diagnostics can look “bad” simply because a bucket has very few events. Confidence intervals help you see whether deviations from the diagonal are meaningful or just noise.
Practical use in scorecards
• Show bucket counts, and optionally error bars.
• If intervals are wide, reduce bucket granularity or increase the evaluation window.
Related
Confidence intervals depend heavily on sample size and help interpret miscalibration signals.