Ask an AI agent to handle the same customer request eight times and you get eight different attempts. Group-relative reinforcement learning, the family of methods that includes GRPO, learns from exactly that spread. It scores every attempt, compares each score with the group's average, and nudges the model toward the attempts that did better and away from the ones that did worse.
Now give the environment a bad day. One attempt runs into a tool outage and cannot finish the job. Another solves the task, but the grader slips and records a failure. Both land below the average, and the update pushes the model away from what they did, although neither did anything wrong. They were unlucky, and the comparison could not tell luck from skill.
Noise itself is not the enemy. Real tools fail, and an agent has to learn to cope with that. The trouble is that each attempt draws its own luck, so every comparison mixes two things: how well the attempt played, and how its dice fell. This post is about a simple remedy, giving every attempt in a group the same luck, and about a question that turned out to be subtler than it looks: when does that actually help? The paper works it out in theory and then tests it on a 2B tool-use agent.
How group-relative RL learns
GRPO1 trains a language model without a separate value network. For each task it samples a group of attempts and scores each one. An attempt's advantage is how far its reward sits above or below the group's average, usually divided by the group's spread. Attempts with a positive advantage are made more likely, and attempts with a negative one less likely.
Comparing attempts at the same task is what makes this work: the task's difficulty is the same for all of them, so it cancels out. Luck does not cancel when every attempt draws its own. The figure shows the worst case, a group in which every attempt solved the task.
One grading slip makes a correct attempt the worst in its group. The update pushes its actions down (−2.47) and nudges the seven others up (+0.35 each), although all eight solved the task.
Same dice for every attempt
The remedy is an old idea from simulation called common random numbers2: when you compare two alternatives, run both on the same random draws, so that any difference comes from the alternatives rather than from the dice. Reinforcement learning has used it before, for example in PEGASUS, which fixes a simulator's random numbers in advance to compare policies.
Applied to GRPO, it gives paired rollouts. All attempts in a group share one noise schedule: the same outages, the same rate limits, the same grading slip. The model's own sampling stays independent, so the attempts still differ in what they do. Every group draws a fresh schedule, so each attempt faces exactly the same odds of bad luck as before. Only its siblings now share that luck.
One detail matters. Attempts take different paths, so “the third tool call” means different things in different attempts. The schedule is therefore keyed by events rather than by call order3: each draw belongs to a tool, the resource it touches and how often that pair has come up before. Two attempts that look up the same order for the first time get the same luck, whenever in the conversation they do it.
| e1 | e2 | e3 | e4 | e5 | e6 | e7 | |
|---|---|---|---|---|---|---|---|
| Rollout 1 | fault | fault | no fault | no fault | fault | no fault | no fault |
| Rollout 2 | fault | no fault | fault | no fault | no fault | no fault | no fault |
| Rollout 3 | no fault | no fault | fault | no fault | no fault | fault | fault |
| Rollout 4 | no fault | fault | no fault | no fault | no fault | no fault | no fault |
Each rollout draws its own luck, so the same event fails for some rollouts and not for others.
Cleaner comparisons, not always a quieter gradient
Before training anything, the paper works out what sharing the dice does to the update.
- No added bias. Luck reaches an attempt only through what the environment shows it, and the update only differentiates the model's own choices. For the standard leave-one-out form of the update, the expected direction is unchanged.
- Cleaner comparisons. Split the variance of a reward into the part explained by luck and the rest, and call the first part's fraction the luck share. With shared dice, the difference between two siblings' rewards loses exactly that fraction of its variance. If luck explains 35% of the reward variance, comparisons get 35% less noisy.
- No guarantee for the gradient. The update multiplies each advantage by a direction in parameter space, the one that makes that attempt's actions more likely, so its noise also depends on how those directions line up. For grading slips, the paper derives the exact condition under which sharing lowers the gradient noise.
For one kind of group, though, sharing can only help: a group whose attempts are all truly correct. Its clean update is zero, so any spread in its rewards is pure grading noise. Independent slips create that spread surprisingly often. Shared slips never do, because either nobody is flipped or everybody is.
At a 10% slip rate, 57% of correct groups look mixed with independent slips, and 0% with shared slips.
| Slip rate | Independent slips | Shared slips |
|---|---|---|
| 2% | 15% | 0% |
| 5% | 34% | 0% |
| 10% | 57% | 0% |
| 10%, measured in training | 55.8% | 0% |
The experiment
To test this on a real agent, I built a simulated back office: customers, orders, products and support tickets behind thirteen tools for looking things up, changing them and waiting out rate limits. Each task is a customer request with one to three parts, such as an address change, a cancellation or a partial refund. A deterministic grader compares the final database with the expected one, field by field.
The environment has two kinds of bad luck, each studied on its own:
- Tool faults. A quarter of tool calls go wrong: a transient error, a rate limit, an outage that lasts for the rest of the episode, a stale read or a truncated list. About three quarters of episodes run into at least one.
- Grader flips. A correct final state is reported as a failure 10% of the time.
The agent is Qwen3.5-2B, trained with GRPO and LoRA for 100 steps of 24 tasks with 8 attempts each. Each design ran on three seeds; seed for seed, the paired and independent runs saw the same tasks in the same order. A reference model was trained without any noise. Every score below is true success: whether the database really ended up right, whatever a noisy grader said. The tests and their pass rules were pre-registered.
- Parameters in the agent
- 2B
- Training runs
- 14
- Episodes per run, with evaluations
- ~27,000
- Seeds per design
- 3
What happened in training
The mechanism behaved as designed. At the start of training, luck explained about a third of the reward variance under tool faults (a luck share of 0.33 to 0.38), so there was plenty for pairing to remove. Under grader flips, 55.8% of all-correct independent groups showed fake disagreement, close to the 57% the formula predicts, against none of the paired groups.
Then the part that matters most: did the agent learn better?
Tool faults: validation
- Training step
- Step 100
- Paired
- 68.0% paired
- Independent
- 63.5% independent
- Trained without noise
- 56.6% trained without noise
| Step | Paired | Independent | Trained without noise |
|---|---|---|---|
| 0 | 54.4% | 52.1% | 51.6% |
| 20 | 60.4% | 56.0% | 55.9% |
| 40 | 60.7% | 57.3% | 55.1% |
| 60 | 67.7% | 58.9% | 57.4% |
| 80 | 64.6% | 66.1% | 55.9% |
| 100 | 68.0% | 63.5% | 56.6% |
Tool faults: recovery challenge
- Training step
- Step 100
- Paired
- 76.6% paired
- Independent
- 69.8% independent
- Trained without noise
- 57.8% trained without noise
| Step | Paired | Independent | Trained without noise |
|---|---|---|---|
| 0 | 55.7% | 56.3% | 54.7% |
| 20 | 58.3% | 64.1% | 53.9% |
| 40 | 67.7% | 64.6% | 57.0% |
| 60 | 71.9% | 67.2% | 57.0% |
| 80 | 76.6% | 68.2% | 58.6% |
| 100 | 76.6% | 69.8% | 57.8% |
Grader flips: clean validation
- Training step
- Step 100
- Paired
- 89.8% paired
- Independent
- 87.8% independent
- Trained without noise
- 89.5% trained without noise
| Step | Paired | Independent | Trained without noise |
|---|---|---|---|
| 0 | 81.8% | 81.8% | 81.3% |
| 20 | 82.5% | 83.6% | 81.6% |
| 40 | 86.5% | 86.7% | 88.3% |
| 60 | 87.5% | 85.9% | 92.6% |
| 80 | 89.8% | 89.8% | 91.0% |
| 100 | 89.8% | 87.8% | 89.5% |
Under tool faults, the paired agent pulled ahead and stayed there. On the final test with faults it reached 66.8% against 61.8%, ahead on all three seeds. Both beat the reference trained without noise, which reached 56.4% on the same test. On the recovery challenge, pairing finished at 72.2% against 68.7%.
Under grader flips there is no clear winner. On the clean test, pairing reached 89.4% against 88.0%, and the reference trained without noise 89.9%.
Trained and tested with tool faults
| Seed 0 | Independent61.4%, paired65.4% | |
|---|---|---|
| Seed 1 | Independent63.1%, paired64.1% | |
| Seed 2 | Independent60.8%, paired71.0% | |
| Mean | Independent61.8%, paired66.8% | |
55%60%65%70%75% |
Trained with grader flips, tested clean
| Seed 0 | Independent90.2%, paired90.5% | |
|---|---|---|
| Seed 1 | Independent85.8%, paired88.5% | |
| Seed 2 | Independent87.9%, paired89.1% | |
| Mean | Independent88.0%, paired89.4% | |
84%86%88%90%92% |
The pre-registered tests give the sober reading. Four of the six passed, including both mechanism checks and no loss on two fault types never seen in training. The two that asked for faster learning did not.
Luck shows up in the rewards
PassNeededA luck share of at least 0.15 under tool faults; under grader flips, fake disagreement near 57% for independent groups and near zero for paired ones
GotLuck share 0.33 to 0.38; fake disagreement 55.8% versus 0%
Gradient noise goes down
PassNeededA lower gradient covariance with pairing at every checkpoint under grader flips, and at most of them under tool faults
GotLower at all eight checkpoints, under both kinds of noise
Faster learning with tool faults
MissNeededArea under the validation curve at least 0.03 higher on average, with every seed ahead
Got+0.037 on average, but one seed was 0.002 behind
Faster learning with grader flips
MissNeededThe same rule, scored on clean validation
Got+0.003, with a 95% interval from −0.029 to +0.033
Better recovery from faults
Pass by its ruleNeededAt least 5 points ahead on the recovery challenge, with every seed ahead
Got+5.4 points and every seed ahead, though the 95% interval (−1.0 to +12.2) includes zero
No loss on unseen fault types
PassNeededNo more than 3 points behind, judged by the lower end of a 90% interval
Got+2.3 points, lower end −0.4
The miss under tool faults is a near miss. The area under the learning curve was 0.037 higher on average, above the 0.03 bar, but the rule asked for every seed to be ahead and one came in 0.002 behind. The miss under grader flips is not close: +0.003, with an interval from −0.029 to +0.033. So the final scores favour pairing under tool faults, but the registered evidence for faster learning is not there.
Did the gradient get quieter?
Reward comparisons and gradient noise can disagree, so I also measured the gradient directly. A probe froze eight checkpoints from two runs trained with tool faults, sampled groups of attempts at each one, and computed every attempt's gradient in full, over all 33.6 million trainable LoRA parameters. It then compared the total variance of the group update under shared and independent luck, on the same attempts.
Grader flips
21 to 30% lower at every checkpoint
| Checkpoint | Reduction |
|---|---|
| Run B, step 0 | 21% |
| Run B, step 20 | 24% |
| Run B, step 40 | 29% |
| Run B, step 60 | 26% |
| Run B, step 80 | 28% |
| Run B, step 100 | 29% |
| Run A, step 80 | 30% |
| Run A, step 100 | 27% |
Tool faults
40 to 63% lower at every checkpoint
| Checkpoint | Reduction |
|---|---|
| Run B, step 0 | 40% |
| Run B, step 20 | 48% |
| Run B, step 40 | 58% |
| Run B, step 60 | 61% |
| Run B, step 80 | 62% |
| Run B, step 100 | 63% |
| Run A, step 80 | 61% |
| Run A, step 100 | 62% |
Pairing lowered the gradient noise at every checkpoint: by 21 to 30% under grader flips, and by 40 to 63% under tool faults. The theory's condition held everywhere too, with room to spare: its left side was only about a fifth to a quarter of its right side.
What to take from it
- If you control the randomness, pairing is cheap. Any simulator, replay layer or grader that exposes its random draws can share them across a group. Each attempt keeps its own odds, and the basic update stays unbiased.
- It has a price. A shared outage can sink a whole group. Under tool faults, 52 to 54% of paired groups gave every attempt the same reward, against about 13% of independent groups, and a group without contrast teaches nothing. A batch of 24 paired groups also sees 24 schedules instead of 192, although the paired agent was not worse on fault types it never saw (+2.3 points).
- Measure the gradient, not just the rewards. The luck share tells you how much of the reward is luck. It is not a validated predictor of learning gains, and the exact condition can fail.
- Keep three claims apart. Cleaner reward comparisons, lower gradient noise and faster learning are different claims. This study supports the first two on its checkpoints, and only part of the third.
Next: luck in agent benchmarks
Luck does not stop at training. Agent benchmarks such as τ-bench4 roll dice of their own: a language model plays the customer, the agent samples its replies, a judge grades some of the conversations, and faults can be injected into the tools. When one agent beats another by a few points, how much of that is skill, and how much is how the dice fell?
The extension I am working on, τ-Dice, carries the keyed dice of paired rollouts over to evaluation. On τ³-bench, every open-model call and every tool fault gets its own seed, derived from what the draw is: which task, which stream, which turn. In principle, any episode can then be replayed, and any one source of luck varied while the others stay fixed. I ran it on the airline, retail and telecom domains with three open agents (Qwen3.5-4B, gpt-oss-20b and Qwen3.6-35B-A3B) and gpt-oss-120b as the customer, plus two more agents for comparison: about 24,500 episodes so far.
- Replays with every seed fixed that repeated the conversation exactly
- 1 of 1,403
- Replays that ended with a different result
- ~3 in 10
- Within-task variation explained by the customer’s or the agent’s seeds
- <5%
- Neighbouring retail leaderboard pairs not separated beyond trial luck
- 7 of 8
The first lesson was humbling: seeds are not enough. Replaying 1,403 episodes with every seed held fixed repeated the conversation word for word exactly once, and about three replays in ten ended with a different result. The inference server adds randomness that no seed controls. Under normal load, the customer's very first message already differed in about three replays out of four. With one episode at a time on the server, that first message almost always matched, but the conversations still drifted apart later, and only 4 of 63 replays per agent repeated exactly.
That noise swamps the dice. The customer's seeds and the agent's seeds each explained under 5% of the variation within a task, and re-running with identical seeds changed outcomes about as often as drawing new ones. Splitting luck by its source will need inference that gives the same answer however busy the server is. The amount of luck, on the other hand, is plain to see: about 60% of the variation in outcomes happens within tasks, so the same agent on the same task often lands on both sides.
Turning the temperature down to 0 does not remove the luck either; it only hides it. Under τ³-bench's official decoding at temperature 0, four identical runs gave different results on 46 to 55% of tasks. The public leaderboard shows the same pattern: in all nine text submissions with four trials each, results changed between trials on 23 to 44% of tasks.
Open agents in τ-Dice, run four times at temperature 0
- Qwen3.5-4B
- 55% , 95% interval 46% to 65%
- gpt-oss-20b
- 47% , 95% interval 37% to 57%
- Qwen3.6-35B-A3B
- 46% , 95% interval 36% to 56%
Public leaderboard submissions, four trials each
- 9 submissions
- 23%–44% : 44%, 44%, 33%, 32%, 30%, 28%, 26%, 24%, 23%
That luck reaches the rankings. With the tasks held fixed and only the trials resampled, 7 of 8 neighbouring pairs on the retail leaderboard are not separated at 95%, and 4 of 8 in airline and in telecom. The judge, at least, adds little luck of its own: grading the same retail conversation four more times changed its verdict in 1.9% of cases.
These results are early. Next comes a blind human check of whether the failures τ-Dice labels as luck look like luck to people, with two annotators judging 200 failures.
References
- 1. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu and Daya Guo (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
- 2. Paul Glasserman and David D. Yao (1992). Some Guidelines and Guarantees for Common Random Numbers. Management Science, 38(6), 884–908.
- 3. Vince Buffalo, Carl A. B. Pearson and Daniel Klein (2026). Realizing Common Random Numbers: Event-Keyed Hashing for Causally Valid Stochastic Models. arXiv:2603.11084.
- 4. Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan (2025). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. International Conference on Learning Representations (ICLR).