Ask a coding agent to fix the same bug eight times and you get eight different patches. Reinforcement learning turns that spread into a lesson: run the tests on every patch, reward the ones that pass, and nudge the model toward them. Group-relative methods such as GRPO1 do exactly this. Each attempt is compared with its siblings, and the ones that beat the group's average are reinforced.
Everything rests on a question that is easy to skip: which tests? A real repository can have thousands. Benchmarks built from GitHub issues, such as SWE-bench and SWE-rebench2, judge a patch by the test files that the issue's original pull request changed, and many training pipelines reuse that check as their reward. We call it the PR's own tests. It is fast, but it only looks where the original fix looked.
A patch can pass the PR's own tests and still break something elsewhere in the repository. We call such a patch a flip. A flip still collects the full reward, so training reinforces the very edit that broke the code. The gap is known from evaluation: an earlier study found that 7.8% of the patches SWE-bench counted as solved, from three agents, fail the repository's full test suite3.
Running the full suite on every attempt would catch flips, but in large repositories it takes tens of times longer. This post is about that trade-off: how often flips happen during training-style sampling, what they do to the learning signal, and how to get the full suite's verdict, on average, while running it on only a small share of attempts.
The PR’s own tests pass six attempts, so each gets +0.25. Attempts 2, 4 and 6 break other tests in the repository, yet training pushes the model toward them exactly as hard as toward the three correct fixes.
Measuring the gap on real agents
Measuring this needs no training run. Nebius has released tens of thousands of trajectories in which Qwen3-Coder-480B, working in the OpenHands agent scaffold, attempts SWE-rebench issues about ten times each. That mirrors what a training step samples: a group of attempts at the same task. For each attempt we rebuilt the task's environment in a cloud sandbox, applied the patch and ran two checks, the PR's own tests and the repository's full suite.
A survey came first. We timed the full suite for one task in each of 1,427 repositories. In most of them it finishes in a few seconds, and the simple answer is to run it. The interesting case is the roughly 15% of repositories with large suites, where the full suite takes about 40 times as long as the PR's own tests. The main study looks there.
- Tasks with large test suites
- 248
- Repositories
- 152
- Attempts that pass the PR’s own tests
- 1,101
- Of those that fail the full suite
- 49
A failure counted only if it happened again when we reran the attempt in fresh sandboxes, and we fixed the hypotheses and analyses before running the study. So: 49 of the 1,101 attempts that passed the PR's own tests failed the full suite. That is 4.5%, about one in 22, and at least 2.4% with 95% confidence. It may sound small. It is not spread evenly.
A few blind spots, many slips
All 49 flips come from just 18 of the 248 tasks, in 17 repositories. When a task has a blind spot, attempts slip through it together: inside those tasks, a median of 27% of passing attempts are flips, and the correlation between attempts in the same group is 0.40.
All 248 tasks with large test suites
18 tasks (7%) hold every one of the 49 flips
Attempts in those 18 tasks that pass the PR’s own tests
49 of 179 fail the full suite
Clustering makes flips matter more than their rate suggests. Measured on groups of eight attempts, the wrong rewards amount to about a quarter of the learning signal, concentrated in about one group in 13. These are systematic errors. The same blind spot pays out whenever its task comes up, so the errors do not average away the way random noise does.
What breaks
Most flips are plain regressions: 28 of the 49 break a test that fails even when it runs on its own. Ten fail only when the whole suite runs, seven of them because of the order in which the tests run, and eleven fail only some of the time.
| Kind of flip | Flips |
|---|---|
| Ordinary regression | 28 |
| Fails only in the full run | 10 |
| Fails only sometimes | 11 |
0102030 |
The full suite is not a perfect judge either. In large suites, about 5% of reruns of attempts that had passed it failed, a reminder that the reference itself carries some noise.
Can smarter test selection fix it?
The classic answer to an expensive suite is regression test selection: run only the tests that a change could affect. We tried three selectors on the same 49 flips. Two pick tests by coverage, the lines or files each test executed before the change. The third is NameRTS4, a recent selector built to be safe: in its own evaluation on human commits, it selected every affected test for 99.6% of them.
| Check | Flips caught (of 49) | Share of full-suite test time |
|---|---|---|
| PR’s own tests | 1 | 7% |
| NameRTS | 20 | 32% |
| Covered lines | 20 | 42% |
| Covered files | 36 | 46% |
| Full suite | 49 | 100% |
None of them closes the gap. The PR's own tests catch 1 of the 49 flips. Selecting by covered lines catches 20, NameRTS 20, and selecting by covered files 36, at nearly half of the full suite's test time. Even the best of them lets about one flip in four through.
Could we at least predict which attempts are risky, and check only those? Not with the obvious signals. Patch size, test coverage of the changed code, suite size and whether the agent ran the tests itself predict flips no better than chance (an AUC of 0.53). A weaker agent, a 70B Llama model in the SWE-agent scaffold, produced flips about as often as the strong one on the same tasks: 7.2% against 7.0%. If you cannot tell which attempts to check, check at random.
Random audits, corrected
Random checking is an old idea from survey sampling. You cannot ask everyone, so you ask a random sample and weight each answer by how unlikely it was to be asked, a trick that goes back to Horvitz and Thompson. For rewards it works in three steps:
- Check every attempt cheaply. Run the PR's own tests on all of them. An attempt that fails them fails the full suite too.
- Audit at random. With a probability fixed in advance, run the full suite on some of the attempts that passed.
- Correct the reward. When an audit finds a flip, scale its correction by one over the audit probability. At a 20% rate, a flip that is found counts five times over.
On average, the corrected signal equals what the full suite would have given, with no bias. Any single step is noisy, because most steps miss and a few correct hard, but the misses and the corrections balance out exactly.
- Draws
- 1000
- Audits so far
- 207
- Running average
- −0.40
Audit the group, not the patch
There is a choice in how to toss the coins. One way gives every attempt its own coin. The other tosses one coin per group and audits all of its passing attempts together. GRPO only uses differences within a group, so an error shared by the whole group cancels out of the update. Flips arrive in clusters, so a whole-group audit reveals the cluster at once and corrects the group's comparisons together.
| Audit rate | Audit attempts one by one, then Audit whole groups | |
|---|---|---|
| Audit rate 10% 4.2 → 2.5 | Audit attempts one by one: 4.2, Audit whole groups: 2.5 | |
| Audit rate 20% 1.9 → 1.1 | Audit attempts one by one: 1.9, Audit whole groups: 1.1 | |
| Audit rate 30% 1.1 → 0.6 | Audit attempts one by one: 1.1, Audit whole groups: 0.6 | |
012345 |
At the same cost, whole-group audits were 42% less noisy than auditing attempts one by one.
Audits can also go where they count. Auditing a group more often when more of its attempts passed the PR's own tests, and when its full suite is cheaper to run, cut the noise by another third at the same cost.
Putting it together
The pieces add up to a simple policy:
- Where the full suite is cheap, as it is in most repositories, run it on every attempt.
- Where it is large, run the PR's own tests on every attempt and audit whole groups at random.
| Reward policy | Test time |
|---|---|
| Always the full suite | 100% |
| Group audits (unbiased) | 23% |
| PR’s tests only (biased) | 10% |
0%50%100% |
Weighted by how common small and large suites are, this gives the full suite's training signal without bias at about 23% of the test time of running everything. The PR's own tests alone take about 10%, but keep the wrong rewards. The audits pay one more dividend: they give a running upper bound on how often the cheap check is wrong, and that bound held in 98% of simulated training runs.
What to take from it
- Which tests run is a reward decision. A check that only looks where the original fix looked will pay for patches that break code elsewhere.
- Wrong rewards are rare but clustered. They come from a few tasks, hit several attempts at once and cannot be predicted from the patch, so they behave like systematic errors rather than noise.
- Selection narrows the gap but does not close it. Random audits with a correction close it, on average, at a fraction of the cost of running everything.
- Audit the group, not the patch. Group-relative training cancels what a group shares, and coupling the audits within a group lets it.
References
- 1. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu and Daya Guo (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
- 2. Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva and Boris Yangel (2025). SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. arXiv:2505.20411.
- 3. You Wang, Michael Pradel and Zhongxin Liu (2025). Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study. arXiv:2503.15223.
- 4. You Wang, Michael Pradel and Zhongxin Liu (2026). Names Are All You Need: Effective and Safe Regression Test Selection for Python. arXiv:2605.25356.