What is calibration error and how would you evaluate alignment of confidence with accuracy in a prompt-based task?

• A. Calibration Error Measures Misalignment Between Predicted Probability And Observed Accuracy; Evaluate With Reliability Diagrams And Brier Score; Temperature Calibration If Needed
• B. Calibration Error Measures Hardware Calibration
• C. Calibration Error Measures Data Distribution Equality
• D. Calibration Error Measures Computation Time

Answer: (C)

Calibration error measures how well a model’s confidence matches real outcomes. In a prompt-based task, you want the probabilities the model outputs to align with actual correctness: a prediction with 0.8 confidence should be correct about 80% of the time.

To evaluate this alignment, reliability diagrams are used. Group predictions into probability bins (like 0.0–0.1, 0.1–0.2, etc.), and for each bin compute the actual accuracy. Plot the predicted probability against the observed accuracy; closer to the diagonal line indicates better calibration. The Brier score provides a single-number summary by taking the mean squared difference between the predicted probabilities and the actual 0/1 outcomes—lower values mean better calibration.

Temperature calibration (or other post-hoc calibration methods) can adjust predicted probabilities after the model is trained to improve alignment, especially when the model is overconfident or underconfident.

The other options miss the concept: hardware calibration concerns sensors, data distribution equality relates to distributional fairness or covariate shift, and computation time is about efficiency, not calibration of probabilities.