Here is a simple experiment. Take an application running on Kubernetes, break it, and hand the cluster to an “agent” that does nothing at all. Then ask the benchmark how the agent did.
I ran this on SREGym, a recent benchmark for AI agents that work as site reliability engineers1. On its missing_service tasks, the answer came back: success. The fault had deleted a Kubernetes Service, and the benchmark's own load generator saw about 10% of user requests fail on one application and 60% on the other. But the grader only checked that the expected pods were running and ready, and the fault injector restarts every pod as its very last step. To the grader, the cluster looked healthy.
Every agent benchmark has a part like this grader, and it rarely gets tested itself. Software testing has a name for it: the test oracle, whatever decides whether an outcome is correct2. When the oracle is wrong, every score built on it is wrong too, and nothing looks out of place. A leaderboard with a broken grader still looks like a leaderboard.
That first finding became a workshop paper, accepted as a poster at the NeurIPS 2026 workshop Who Verifies the Agents?. I have since extended it from one benchmark to four, with more kinds of wrong fixes made on purpose, a stopwatch on the grader and three real AI agents. The extended paper is under review at ICST, and this post walks through what it found.
How these benchmarks grade an agent
Live SRE benchmarks broadly run the same loop. They deploy a real application, such as a hotel booking service, a social network or an online shop built from microservices, on a Kubernetes cluster. They inject a fault and hand the cluster to an agent along with a short task description. When the agent is done, a grading script checks the cluster, directly or through its monitoring, and returns pass or fail.
- Deploy A real application on a Kubernetes cluster
- Break The benchmark injects a fault
- Repair An AI agent investigates and acts
- Grade A script checks the cluster: pass or fail
| Graded pass | Graded fail | |
|---|---|---|
| The fix worked | Right Credit for a real fix | False reject A real fix loses its credit |
| Still broken | False accept Unearned credit | Right No credit, as it should be |
A grader can be wrong in two directions. A false accept passes a cluster that is still broken, so the agent gets credit it did not earn. A false reject fails a cluster that was fixed, so a correct repair loses its credit. Both move scores, and neither shows up in the score itself.
Eight questions for a grader
One bug in one task is an anecdote. To test graders systematically, I borrowed what software testing already knows about oracles, mutation testing, flaky tests and test isolation, and turned it into eight properties. Each one is a question a good grader should answer with yes, and each has a live experiment behind it.
- Does it fail a broken cluster? An agent that does nothing should never pass.
- Does it pass a healthy cluster? If nothing is wrong, there is nothing to fail.
- Does it pass a correct fix? Both the benchmark's own reference fix and other fixes that also work.
- Does it fail wrong fixes? I applied plausible but wrong repairs on purpose, as you will see below.
- Is its verdict stable? The same action should get the same verdict, however often you ask and whenever you look.
- Does it ignore other tasks? Leftovers from one task should not change another task's verdict.
- Does the fix have to last? A repair that holds for a minute and then breaks again should not pass.
- Is the task itself sound? The fault should take effect, the task description should point at it, and the answer should not be lying around in the cluster.
I ran these experiments on four open benchmarks: SREGym, AIOpsLab, k8s-ai-bench and ITBench. Each was pinned at a fixed version (SREGym also at a newer one), and the runs used live clusters rather than recorded traces.
- Benchmarks audited
- 4
- Live runs
- 1,269
- Of them, runs by three AI agents
- 182
- Tasks with at least one grading defect
- 140
| A good grader… | SREGym | AIOpsLab | k8s-ai-bench | ITBench |
|---|---|---|---|---|
| Fails a broken cluster | 3/124 | 4/14 | 0/22 | 2/23 |
| Passes a healthy cluster | 28/119 | 0/14 | not measured | 3/23 |
| Passes a correct fix | 17/124 | 0/14 | 6/9 | 2/9 |
| Fails wrong fixes | 53/109 | 13/13 | 3/13 | 7/23 |
| Gives a stable verdict | 28/124 | 12/14 | 3/22 | 1/7 |
| Ignores other tasks | 14/125 | not measured | 1/1 | not measured |
| Requires the fix to last | 1/1 | not measured | not measured | 16/23 |
| Comes with a sound task | 1/124 | 5/14 | 1/24 | 4/25 |
| Properties broken | 8 of 8 | 4 of 8 | 5 of 8 | 7 of 8 |
Every benchmark breaks at least four of the eight properties, and SREGym, the largest, breaks all eight. The weak spots differ. SREGym and AIOpsLab let wrong fixes through: 53 of 109 SREGym tasks and all 13 AIOpsLab tasks I tested accepted at least one. AIOpsLab's verdicts also depend on timing: in 12 of its 14 tasks, the same action passed or failed depending on when the grader looked. k8s-ai-bench rejects correct fixes that differ from the one it expects, in 6 of 9 tasks tested, and ITBench passes fixes that do not last, in 16 of 23.
The usual checks miss most of it
Careful benchmark builders already run two sanity checks: an agent that does nothing must fail, and the benchmark's own reference fix must pass. Recent guidance on rigorous agent benchmarks recommends the first one explicitly3. Both are worth running, and in this study they caught problems in 22 of the 140 tasks with a defect. The other 118, or 84%, have defects that only the other experiments reveal.
| Benchmark | Caught and missed | Missed, of all tasks with a defect |
|---|---|---|
| SREGym | 14 caught, 79 missed | 79 of 93 |
| AIOpsLab | 4 caught, 10 missed | 10 of 14 |
| k8s-ai-bench | 0 caught, 10 missed | 10 of 10 |
| ITBench | 4 caught, 19 missed | 19 of 23 |
| All four | 22 caught, 118 missed | 118 of 140 |
The two checks test the two ends of a task: nothing done, or everything done right. Most of what they miss lives in between, in the wrong fix, the early verdict, the leftovers of the previous task and the fix that quietly breaks again.
Wrong fixes, on purpose
The most revealing experiment borrows from mutation testing. Instead of waiting for an agent to make a mistake, I made the mistakes myself. Each wrong fix, or mutant, is something a hurried engineer or a confused agent could plausibly do:
- Restart: delete every pod and let them come back, without touching the cause.
- Scale to zero or delete: make the broken workload go away instead of fixing it.
- Collateral damage: apply the correct fix, then scale an unrelated, healthy service down to zero.
A good grader rejects all of them. SREGym's graders caught almost every restart, scale to zero and delete, 160 of 162. But they accepted the collateral damage in 49 of 96 tasks, because each grader only checks the part of the system the fault touched.
AIOpsLab's grader did worse. It waits until the pods it can see are healthy, and a deleted or scaled-down workload has no pods left to be unhealthy. Under its own rules, it accepted scale to zero and delete in 5 of 6 tasks each, and collateral damage in all 13. The fix I proposed is small: remember which workloads the application deployed, and fail while one of them is missing or scaled to zero. With it, the grader caught 24 more wrong fixes and let through none that it had caught before.
| Wrong fix | AIOpsLab's own check, then With the workload check | |
|---|---|---|
| Restart 83% → 92% | AIOpsLab's own check: 83%, With the workload check: 92% | |
| Scale to zero 17% → 100% | AIOpsLab's own check: 17%, With the workload check: 100% | |
| Delete 17% → 100% | AIOpsLab's own check: 17%, With the workload check: 100% | |
| Collateral damage 0% → 100% | AIOpsLab's own check: 0%, With the workload check: 100% | |
0%25%50%75%100% |
When the grader looks matters
Some graders do not look at the cluster's state directly. They ask the monitoring system whether any alerts are still firing. That sounds reasonable, but alerts lag behind the system. SREGym's alert grader, in a newer version than the one I pinned, waits 30 seconds after the agent finishes, fails at the first firing alert, and passes only after two minutes with none. But the error-rate alert is computed over the last two minutes of requests and re-evaluated once a minute, so it keeps firing for a while after the errors stop.
I measured what that does on eight problems, after three kinds of action: SREGym's own reference fix, no fix at all, and a restart only. Judged right away, the reference fix failed on all eight. Allowed to wait up to five minutes, the grader would pass three; up to ten minutes, all eight. The correct fixes needed between 211 and 573 seconds to look correct.
SREGym’s own fix
0 right away, 3 within 5 min, 8 within 10
No fix
0 right away, 1 within 5 min, 1 within 10
Restart only
0 right away, 0 within 5 min, 1 within 10
Waiting is not free, though. Given ten minutes, the grader also passed one problem with no fix at all, and the same problem after only a restart: its alerts went quiet on their own while the fault was still in place. A grader that waits should also check that the fault itself is gone.
This is not a lab curiosity. When real agents correctly fixed five of SREGym's alert problems, the grader failed 26 of their 45 fixes, and in every one of those 26 the same alert, HighRequestErrorRate, was still firing when it looked. The SREGym maintainers welcomed the report and invited a fix, which is now an open pull request.
Real agents, real misgrades
Mutants are controlled mistakes. To see what graders do with real behavior, I ran three agents on tasks from SREGym, AIOpsLab and ITBench: Claude Code with Claude Sonnet 5.5 and with Claude Haiku 5.5, and OpenAI's Codex CLI with gpt-6.1-sol, 182 runs in all. I compared each verdict with an independent reference check that looks at what users would notice: whether requests succeed, whether services still route traffic, and whether capacity is back to the healthy baseline.
- AIOpsLab: 1 of 56 verdicts wrong
- 2%
- ITBench: 22 of 45 wrong
- 49%
- SREGym, pinned: 10 of 23 wrong
- 43%
- SREGym, newer: 35 of 54 wrong
- 65%
How often the verdict disagreed with the reference depends on the benchmark, not on the agent. On the tasks they shared, Sonnet, Haiku and Codex were misgraded about equally often: an exact McNemar test finds no difference for any pair (p = 1.0).
| Sonnet | Haiku | Codex | |||||||
|---|---|---|---|---|---|---|---|---|---|
| SREGym, newer version | |||||||||
| Ad service failure | Sonnet, attempt 1: Graded pass, and the fix worked | Sonnet, attempt 2: Graded fail, but the fix had worked | Sonnet, attempt 3: Graded pass, and the fix worked | Haiku, attempt 1: Graded pass, and the fix worked | Haiku, attempt 2: Graded fail, but the fix had worked | Haiku, attempt 3: Graded fail, but the fix had worked | Codex, attempt 1: Graded fail, but the fix had worked | Codex, attempt 2: Graded pass, and the fix worked | Codex, attempt 3: Graded fail, but the fix had worked |
| Ad service high CPU | Sonnet, attempt 1: Graded pass, and the fix worked | Sonnet, attempt 2: Graded pass, and the fix worked | Sonnet, attempt 3: Graded fail, but the fix had worked | Haiku, attempt 1: Graded fail, but the fix had worked | Haiku, attempt 2: Graded pass, and the fix worked | Haiku, attempt 3: Graded pass, and the fix worked | Codex, attempt 1: Graded fail, but the fix had worked | Codex, attempt 2: Graded pass, and the fix worked | Codex, attempt 3: Graded fail, but the fix had worked |
| Cart service failure | Sonnet, attempt 1: Graded pass, and the fix worked | Sonnet, attempt 2: Graded fail, but the fix had worked | Sonnet, attempt 3: Graded fail, but the fix had worked | Haiku, attempt 1: Graded pass, and the fix worked | Haiku, attempt 2: Graded fail, but the fix had worked | Haiku, attempt 3: Graded pass, and the fix worked | Codex, attempt 1: Graded pass, and the fix worked | Codex, attempt 2: Graded fail, but the fix had worked | Codex, attempt 3: Graded fail, but the fix had worked |
| Payment service failure | Sonnet, attempt 1: Graded fail, but the fix had worked | Sonnet, attempt 2: Graded fail, but the fix had worked | Sonnet, attempt 3: Graded fail, but the fix had worked | Haiku, attempt 1: Graded pass, and the fix worked | Haiku, attempt 2: Graded fail, but the fix had worked | Haiku, attempt 3: Graded pass, and the fix worked | Codex, attempt 1: Graded pass, and the fix worked | Codex, attempt 2: Graded pass, and the fix worked | Codex, attempt 3: Graded pass, and the fix worked |
| Product catalog failure | Sonnet, attempt 1: Graded pass, and the fix worked | Sonnet, attempt 2: Graded fail, but the fix had worked | Sonnet, attempt 3: Graded fail, but the fix had worked | Haiku, attempt 1: Graded fail, but the fix had worked | Haiku, attempt 2: Graded fail, but the fix had worked | Haiku, attempt 3: Graded fail, but the fix had worked | Codex, attempt 1: Graded fail, but the fix had worked | Codex, attempt 2: Graded fail, but the fix had worked | Codex, attempt 3: Graded fail, but the fix had worked |
| Wrong operator image | Sonnet, attempt 1: Graded pass, but the cluster was still broken | Sonnet, attempt 2: Graded pass, but the cluster was still broken | Sonnet, attempt 3: Graded pass, but the cluster was still broken | Haiku, attempt 1: Graded pass, but the cluster was still broken | Haiku, attempt 2: Graded pass, but the cluster was still broken | Haiku, attempt 3: Graded pass, but the cluster was still broken | Codex, attempt 1: Graded pass, but the cluster was still broken | Codex, attempt 2: Graded pass, but the cluster was still broken | Codex, attempt 3: Graded pass, but the cluster was still broken |
| ITBench | |||||||||
| Scenario 114 | Sonnet, attempt 1: Graded pass, but the cluster was still broken | Sonnet, attempt 2: Graded pass, but the cluster was still broken | Sonnet, attempt 3: Graded pass, but the cluster was still broken | Haiku, attempt 1: Graded pass, but the cluster was still broken | Haiku, attempt 2: Graded pass, but the cluster was still broken | Haiku, attempt 3: Graded pass, and the fix worked | Codex, attempt 1: Graded pass, but the cluster was still broken; the agent edited the alert rules | Codex, attempt 2: Graded pass, but the cluster was still broken; the agent edited the alert rules and the telemetry collector | Codex, attempt 3: Graded pass, but the cluster was still broken |
| Scenario 31 | Sonnet, attempt 1: Graded fail, but the fix had worked | Sonnet, attempt 2: Graded fail, but the fix had worked | Sonnet, attempt 3: Graded fail, but the fix had worked | Haiku, attempt 1: Graded fail, but the fix had worked | Haiku, attempt 2: Graded pass, but the cluster was still broken; the agent changed the telemetry collector | Haiku, attempt 3: Graded fail, but the fix had worked | Codex, attempt 1: Graded pass, and the fix worked; the agent edited the alert rules | Codex, attempt 2: Graded pass, and the fix worked; the agent edited the alert rules | Codex, attempt 3: Graded pass, and the fix worked; the agent edited the alert rules |
| Scenario 58 | Sonnet, attempt 1: Graded fail, but the fix had worked | Sonnet, attempt 2: Graded fail, but the fix had worked | Sonnet, attempt 3: Graded fail, but the fix had worked | Haiku, attempt 1: Graded fail, but the fix had worked | Haiku, attempt 2: Graded pass, and the fix worked; the agent changed the telemetry collector | Haiku, attempt 3: Graded fail, but the fix had worked | Codex, attempt 1: Graded fail, but the fix had worked | Codex, attempt 2: Graded fail, but the fix had worked | Codex, attempt 3: Graded fail, but the fix had worked |
The grid shows both kinds of error side by side. On SREGym's alert problems, the crosses are correct fixes failed by the timing problem above. On ITBench's scenario 114, every run passed, yet in eight of the nine the product catalog service still could not be reached through its Service. On scenarios 31 and 58, a correct fix failed unless the agent had also changed the monitoring: the only runs that passed were ones in which it had. Three stories stand out.
The operator nobody touched
SREGym's wrong operator image task breaks the operator that manages a TiDB database, in one namespace. The grader checks a second namespace, and the task description points the agent at a third, where the application lives. All nine runs passed with the operator still broken, and not one agent ran a single command in the operator's namespace. “I found nothing broken in the FleetCast app, so I changed nothing,” one of them reported. The maintainers are now fixing this check in a pull request of their own.
The rules that moved
ITBench passes a run once the expected alerts stop firing. In 9 of its 16 ITBench runs, Codex edited the alerting rules themselves, the very rules that check reads, and the check passed all 9. Neither Claude model touched the rules, although Haiku changed the telemetry collector in three runs. Some of Codex's runs also fixed the fault and some did not. Either way, a grader that reads something the agent can edit cannot tell the difference.
The answer key in the room
AIOpsLab leaves mitigation scripts inside the database pods of four of its problems. The agents found them: in 10 runs, an agent ran a script that the benchmark had left behind. It is the benchmark version of an exam with the answer key on the desk.
Does it move the leaderboard?
Sometimes a lot, sometimes not at all. k8s-ai-bench publishes its raw results, so instead of running agents on it I re-scored its published leaderboard: 1,200 runs of ten models on 24 tasks. I credited runs that a check rejected only because the container image was not written as the exact string “nginx”, and set aside failures caused by the benchmark or its harness, such as a check that runs before its database pod is scheduled.
| Model | As published, then Re-scored | |
|---|---|---|
| gemini-3-pro-preview 95.8% → 95.8% | As published: 95.8%, Re-scored: 95.8% | |
| gemini-2.5-pro 95% → 95% | As published: 95%, Re-scored: 95% | |
| GLM-4.5 91.7% → 92.4% | As published: 91.7%, Re-scored: 92.4% | |
| Qwen3-Coder-480B 89.2% → 89.2% | As published: 89.2%, Re-scored: 89.2% | |
| GLM-4.6 89.2% → 89.2% | As published: 89.2%, Re-scored: 89.2% | |
| Qwen3-Next-80B 84.2% → 84.2% | As published: 84.2%, Re-scored: 84.2% | |
| gemini-2.5-flash 81.7% → 81.7% | As published: 81.7%, Re-scored: 81.7% | |
| Qwen3-Coder-30B 66.7% → 67.2% | As published: 66.7%, Re-scored: 67.2% | |
| gpt-oss-120b 62.5% → 67.5% | As published: 62.5%, Re-scored: 67.5% | |
| gpt-oss-20b 40.8% → 51.9% | As published: 40.8%, Re-scored: 51.9% | |
0%25%50%75%100% |
Most scores did not move, and the ranking barely changed: one pair of neighbors swapped places (Kendall's τ = 0.956). But gpt-oss-20b rose from 40.8% to 51.9%, 11 points from grading alone. Grader errors do not spread evenly across agents, so a few points between two agents can be the grader's doing rather than theirs.
If you build or use these benchmarks
The fixes are mostly cheap. A short checklist for the graders of live benchmarks:
- Keep the two usual checks, the agent that does nothing and the reference fix, and run them on every task.
- Throw wrong fixes at the grader. A restart, a deletion, scaling to zero and a correct fix with collateral damage should all fail.
- Judge after the system settles, within a stated time limit, and also check that the fault itself is gone.
- Compare against the healthy state of the whole application, not only the part the fault touched.
- Isolate tasks, or verify the reset between them.
- Check that fixes last, for example when a pod is replaced.
- Keep the answer and the grader's inputs, such as alert rules, out of the agent's reach.
And if you only use these benchmarks, read a score together with how its grader was tested. I reported the problems to the maintainers of SREGym, AIOpsLab and ITBench, with proposed fixes for AIOpsLab and ITBench, and SREGym's two fixes are underway as described above.
References
- 1. Jackson Clark, Yiming Su, Saad Mohammad Rafid Pial, Yifang Tian, Lily Gniedziejko, Hans-Arno Jacobsen, Yinfang Chen and Tianyin Xu (2026). SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios. arXiv:2605.07161.
- 2. Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz and Shin Yoo (2015). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41(5), 507–525.
- 3. Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor et al. (2025). Establishing Best Practices for Building Rigorous Agentic Benchmarks. arXiv:2507.02825.