Paper
16 min
Checking your own work
Self-verification at scale: why running the code, re-deriving the maths and arguing the other side catches most mistakes.
J. Okafor, M. Castell
A model that can check its answer can afford to be adventurous about finding one. We trained verifiers alongside the model and let them veto answers.
Three checks catch 88% of wrong answers before they reach the user: executing code, re-deriving results a second way, and generating the strongest counter-argument.
The three checks
Run it: execute code and read the errors
Re-derive it: solve again by a different route
Argue it: write the best case against the answer
Checks add about 30% to thinking time. On hard problems the trade is almost always worth it.
Keep reading
More from the lab.
System card
32 min
Kerrfield One system card
How Kerrfield One was trained, what 11,400 hours of red-teaming found, and every mitigation we shipped before launch.
Paper
18 min
Thinking longer, not bigger
A 70B model given four minutes to think beats a model ten times its size answering at once. Here is why, and where it stops working.
Paper
14 min
Calibration you can check
When Kerrfield One says it is 80% sure, it is right about 80% of the time. How we trained for honest confidence, and how to verify it yourself.