Paper
14 min
Calibration you can check
When Kerrfield One says it is 80% sure, it is right about 80% of the time. How we trained for honest confidence, and how to verify it yourself.
A. Moreau, S. Iyer
A confident wrong answer is worse than no answer. We trained Kerrfield One to report how likely it is to be right, and to say so plainly when that number is low.
Across 40,000 held-out questions, the model’s stated confidence is within 3.1 points of its real accuracy. The previous model was off by 7.8 points.
How to check it
Collect answers with their confidence scores
Bucket them in steps of ten points
Compare stated confidence with accuracy in each bucket
We include a notebook that does this against your own data in about five minutes.
Keep reading
More from the lab.
System card
32 min
Kerrfield One system card
How Kerrfield One was trained, what 11,400 hours of red-teaming found, and every mitigation we shipped before launch.
Paper
18 min
Thinking longer, not bigger
A 70B model given four minutes to think beats a model ten times its size answering at once. Here is why, and where it stops working.
Paper
14 min
Calibration you can check
When Kerrfield One says it is 80% sure, it is right about 80% of the time. How we trained for honest confidence, and how to verify it yourself.