Calibration Drift
Calibration drift is a change in calibration over time, where probability buckets no longer match observed frequencies. It often follows base-rate shifts or regime changes.
Definition
Calibration drift happens when a previously calibrated forecasting process becomes miscalibrated over time. For example, your 70% bucket used to resolve near 70% but now resolves near 55%.
Why it happens
• Changes in question mix or difficulty
How to detect it
• Track rolling window calibration curves.
• Monitor bucket realized frequencies over time.
Related
Calibration drift is a specific form of forecast drift and is reflected in miscalibration.