arXiv cs.CLPaper
Calibration as a First-Class Criterion in LLM Evaluation
This is a practical call to rigor. If you're using LLMs for anything with real consequences, you need calibrated confidence scores. The paper points out that methods like LLM-as-a-judge, synthetic data generation, and active learning all assume calibration without measuring it. Most benchmarks have the inputs needed to compute calibration metrics right now. If you're evaluating a new model or method, add this to your checklist before you ship.