Engineering
12 min
Two million tokens, one question
How we made a two-million-token context window useful rather than just long: retrieval inside the model, and tests that catch forgetting.
L. Brandt, R. Nakamura
Most long-context models can accept a huge input. Fewer can use it. We measured what gets lost and rebuilt attention so the model finds the one line that matters.
On our retrieval test, Kerrfield One finds a planted fact anywhere in two million tokens 97.2% of the time, and it can explain why that fact answers the question.
What changed
Chunked attention with learned routing
Training on tasks that require long-range links
A forgetting test we now run on every checkpoint
The retrieval test is published so other labs can report the same number.
Keep reading
More from the lab.
System card
32 min
Kerrfield One system card
How Kerrfield One was trained, what 11,400 hours of red-teaming found, and every mitigation we shipped before launch.
Paper
18 min
Thinking longer, not bigger
A 70B model given four minutes to think beats a model ten times its size answering at once. Here is why, and where it stops working.
Paper
14 min
Calibration you can check
When Kerrfield One says it is 80% sure, it is right about 80% of the time. How we trained for honest confidence, and how to verify it yourself.