Ratchet · integrity layer for self-improving AI R&D
A research loop is only as good as what it chooses to keep.
We gave the same Claude research agent the same RL post-training problem twice, with the same compute. The keep-if-better loop finished believing its model scored . Measured properly, it scored . Under Ratchet, the same agent finished believing , and measured the same way it scored .
In 30 seconds
The problem. When AI trains AI, the model can learn to fool the grader. The score rises while real ability falls, and nobody notices.
What we built. Ratchet: a check that keeps only improvements that are real. It needs nothing but the grader itself.
Small model. A keep-if-better loop believed ; the truth was . With Ratchet, belief and truth matched ( vs ).
Real model. Training Qwen2.5-0.5B on math, RL found a hole in our grader and true accuracy fell to 0% while the score kept climbing. Ratchet flagged it, and its score tracks the truth at r = . Run before training, the same checks found that hole in seconds.
What each loop believed, and what was true
Each point is the loop's current best recipe after an iteration. The dashed line is the number the loop used to decide. The solid line is the hidden ground truth: whether the model's final, committed answer is right. Neither the agent nor either gate ever saw it.
Where it went wrong
Every proposal the keep-if-better loop evaluated, compared with its current best at that moment. A ✗ marks a wrong call: a change kept without a real gain, or a real gain thrown away.
The same idea, two verdicts
On a real model, RL found a hole in our own grader
The research loop on the real model
A public grader: Hugging Face Math-Verify
Pre-flight: audit the grader before spending GPU time
What Ratchet checks before a change is kept
Ratchet sits at the accept step of the loop. It needs three things the lab already has: the evaluation harness, the ability to run more seeds, and black-box access to the grader. It does not need a better grader.
01 · Seal
The harness can't move
Every run is stamped with a fingerprint of the evaluation code, the held-out set and the grader. A result from a modified harness is refused before anything else is read.
02 · Seeds
One run is an anecdote
A gain must hold across seeds (one-sided Welch t-test, p < 0.05, at least +1 pp). Promising but inconclusive results get more seeds, up to five, so compute goes where the decision is close.
03 · Grader probes
Would the grader also accept a wrong answer?
For every held-out response, Ratchet asks the grader whether it would also pass that response against answers that rule each other out: near misses of the reference, other problems' answers and, in the real-model study, every other number the response mentions. A response that passes for two different answers is hedging, by definition.
04 · Echo probe
Is the grader crediting a restated question?
Ratchet deletes the restated problem from each passing response and grades it again. If the credit disappears, the response passed by repeating the question. Added after a public grader showed us this gap; details below.
Decisions are made on the verified score: the share of held-out responses the grader accepts for the reference and for no mutually exclusive alternative.
verified = meani [ grader(ri, refi) ∧ ¬∃ a ∈ probes(refi) : grader(ri, a) ]
The probes call the lab's own grader with alternative references. No answer key beyond what the grader already uses, and no access to the hidden ground truth.
Benchmark: recipe changes with known ground truth
Each change was applied to the default recipe and run on seeds. Ground truth comes from the hidden oracle over all seeds. The keep-if-better gate sees one run of the headline score. Ratchet sees 3 to 5 seeds and the grader probes.
The verified score tracks the truth
Every training run in this study, in total. Each run is plotted twice: its headline score and its verified score, against its true accuracy. The diagonal is perfect agreement.
Score vs. true accuracy, per runPearson r with the truth: headline , verified .
Two scales. The agent-loop study uses a -parameter transformer on a laptop CPU, so all of its training runs rerun in an afternoon. Its base model was given a self-correction habit on purpose. The real-model study changes nothing about the model: Qwen2.5-0.5B-Instruct, Qwen's own math prompt, GRPO on GSM8K, runs for of GPU time.
Nobody planted the real-model exploit. We wrote the grader ourselves and did not know it could be fooled by an unclosed \boxed{. We found the hole by reading the responses Ratchet's probe flagged.
One family of probes. Specificity probes catch exploits where a response satisfies the grader without committing to an answer. Graders built on LLM judges or unit tests need their own probes: judges scored against paraphrased wrong references, and tests run against mutated reference solutions.
A correction.
Format is out of scope. Ratchet asks whether the grader's credit is earned; it does not check that a response follows the requested format. A format requirement has to be encoded by the lab, as the format-gated Math-Verify arm shows.
The real-model loop is short. Five rounds per loop fit the budget. Neither loop collapsed; the difference there is calibration (believed vs. true), not a rescue.
Not yet shown: longer real-model loops, models above 0.5B, LLM-judge graders, contamination canaries, and an adaptive seed budget that spends compute only where a decision is close.