Open a pull request on a large project and its CI pipeline will usually run the entire test suite, whether you rewrote a module or fixed a typo in a comment. On the projects in this study, one full run takes anywhere from half an hour to 40 hours of test time. Multiply that by every change, every day, and you get slow feedback, large cloud bills and a real energy footprint.
The obvious fix is test selection: run only the tests that matter for this change and skip the rest. It is also the subject of one of the industry challenges posed for ICSE 2027, Enhancing CI/CD Pipeline Cost-Effectiveness through Commit-Aware Test Selection. The challenge asks for selection that is safe enough for production, stays accurate as a code base keeps changing, and saves more than it costs to run. It also poses an open question: can reading what a change actually does, rather than only where it sits, make selection both accurate and safe?
While working on that challenge, I ran into a problem one step earlier. Before we can say which selector is best, we have to measure them fairly, and the usual way of measuring them quietly lets a selector see things it could never see in production. This post is about what happens when you take that away.
How test selectors are graded
Test selectors are evaluated by replay. You take a project's recorded CI history, walk through it build by build, let the selector choose which tests to run, and check whether it would have caught the failures that really happened.
A large recent benchmark of this kind is LRTS, a dataset of long-running test suites from ten large open-source projects1. Its authors compared many ways of ordering tests and found that a simple one wins: “prioritizing faster tests that recently failed performs the best, outperforming the sophisticated techniques.” In this post I call that order recent failures first.
There is a catch. LRTS studies prioritization: every test still runs, just in a smarter order, so every result is legitimately known afterwards. A selector skips tests, and a skipped test produces no result. In production, a selector never learns that a test it skipped would have failed. Statisticians call this censoring. A replay that keeps feeding the selector every result lets it peek. There is a second, smaller leak about time: a replay may also show results from builds that had started but not yet finished.
| Build1 | Build2 | Build3 | Build4 | Build5 | Build6 | Build7 | Build8 | |
|---|---|---|---|---|---|---|---|---|
| T1, Test 1 | ||||||||
| T2, Test 2 | ||||||||
| T3, Test 3 | ||||||||
| T4, Test 4 | ||||||||
| T5, Test 5 | ||||||||
| T6, Test 6 |
Full replay: 3 failures visible.
| Build1 | Build2 | Build3 | Build4 | Build5 | Build6 | Build7 | Build8 | |
|---|---|---|---|---|---|---|---|---|
| T1, Test 1 | ||||||||
| T2, Test 2 | ||||||||
| T3, Test 3 | ||||||||
| T4, Test 4 | ||||||||
| T5, Test 5 | ||||||||
| T6, Test 6 |
Finished builds only: 2 failures visible.
| Build1 | Build2 | Build3 | Build4 | Build5 | Build6 | Build7 | Build8 | |
|---|---|---|---|---|---|---|---|---|
| T1, Test 1 | ||||||||
| T2, Test 2 | ||||||||
| T3, Test 3 | ||||||||
| T4, Test 4 | ||||||||
| T5, Test 5 | ||||||||
| T6, Test 6 |
Faithful replay: 1 failures visible.
Builds 7 and 8 are still running when the selector picks tests for build 9. Failures it can see: 3 in the full replay, 2 with finished builds only, 1 in the faithful replay.
Recent failures first depends on exactly the information that censoring removes. If it skips a test that then starts failing, it never sees the failure, so it keeps skipping the test. The full replay tells it anyway.
Replaying honestly
So I built a replay that only shows a selector what it could have known: results from finished builds, and only for the tests it ran. I call it the faithful replay and compared it with the usual full one, and with one that hides only the unfinished builds, so that the two leaks can be told apart.
- Open-source projects (Kafka, Hadoop, Hive, …)
- 10
- Builds in the LRTS dataset
- 21,255
- Newer builds from the same projects
- 13,389
- Industrial CI logs, from ABB and Google
- 3
The measure is simple. A build with a real failure escapes when the selection ran none of its failing tests; failures of tests already known to be flaky do not count. To compare selectors fairly, every comparison runs them at the same budget, 20% of the suite's test time.
I also wanted the results to be hard to argue with, so I wrote down every hypothesis and analysis before evaluating any held-out build, in two stages. Everything was tuned on older builds and then tested twice: on the last stretch of the LRTS history, and on the newer builds that came after it.
History pays, the change does not
| Order | Full replay, then Faithful replay | |
|---|---|---|
| Recent failures first 10.2% → 30.5% | Full replay: 10.2%, Faithful replay: 30.5% | |
| DANTE 12.0% → 28.0% | Full replay: 12.0%, Faithful replay: 28.0% | |
| Hybrid: failures + change 11.6% → 22.5% | Full replay: 11.6%, Faithful replay: 22.5% | |
| Change match, cheapest first 32.6% → 35.6% | Full replay: 32.6%, Faithful replay: 35.6% | |
| Change match 39.9% → 40.4% | Full replay: 39.9%, Faithful replay: 40.4% | |
0%10%20%30%40%50% |
Orders that rank tests by their failure history pay heavily. Recent failures first lets 10.2% of failing builds escape under the full replay and 30.5% under the faithful one, three times as many. DANTE, a recent data-driven method evaluated on the same dataset that weighs failure history together with the change, goes from 12.0% to 28.0%.
Orders that read the change itself barely notice. The change match ranks tests by how closely their source text matches the change's diff, with the text-retrieval scoring known as TF-IDF. It escapes 39.9% and 40.4% under the two replays. Its cost-aware version pays a little, because even a test's run time is history: when a test is skipped, its last known duration goes stale.
The leak about time turned out to be small: hiding only the unfinished builds changes every escape rate in these figures by less than 5%. Almost the entire price comes from censoring.
| Order | Held-out builds, then Newest builds | |
|---|---|---|
| Recent failures first 1.32× → 2.98× | Held-out builds: 1.32×, Newest builds: 2.98× | |
| DANTE 1.45× → 2.34× | Held-out builds: 1.45×, Newest builds: 2.34× | |
| Hybrid: failures + change 1.25× → 1.93× | Held-out builds: 1.25×, Newest builds: 1.93× | |
| Change match, cheapest first 1.12× → 1.09× | Held-out builds: 1.12×, Newest builds: 1.09× | |
| Change match 0.97× → 1.01× | Held-out builds: 0.97×, Newest builds: 1.01× | |
1×2×3× |
The price also grows. On the held-out end of the LRTS history, recent failures first missed a third more failing builds under the faithful replay (1.32×). On the newer builds it missed nearly three times as many (2.98×). One plausible reason: the further a selector runs on its own choices, the staler its picture of the tests it keeps skipping. Selectors that read the change stay near 1×.
Industrial logs say the same
Open-source projects might not behave like company pipelines, so I also replayed three industrial CI logs that are widely used in research: two from ABB Robotics and one from Google, in the form released with the Retecs study2. The logs ship each test's earlier results as a precomputed column, and the Retecs tooling hands that column to the selector as it is. An evaluation built on that tooling therefore runs with full history, including the results of tests the selector skipped.
| Log | Full history, then Faithful history | |
|---|---|---|
| Google GSDTSR 17.5% → 21.3% | Full history: 17.5%, Faithful history: 21.3% | |
| ABB Paint Control 72.0% → 75.2% | Full history: 72.0%, Faithful history: 75.2% | |
| ABB IOF/ROL 68.3% → 67.4% | Full history: 68.3%, Faithful history: 67.4% | |
0%25%50%75%100% |
On Google's log, faithful history raises the share of missed failing verdicts from 17.5% to 21.3%. On the two ABB logs a 20% budget misses most failures either way, which leaves little room for a difference. Across the three logs, the misses still rise by 3.9% (95% interval 1.2% to 7.9%).
The ranking can flip
A price paid by every selector alike would be bad news, but it would not change which selector to pick. The trouble is that selectors pay very different prices, so the ranking itself can change.
Industry often learns selection from data. Facebook's predictive test selection, for example, trains a model on historical test outcomes and halved the cost of testing changes in production3. I built a learned selector in that spirit: a gradient-boosted model that combines change relevance, failure history, staleness and run time, and keeps retraining on the outcomes it observes.
| Order | Full replay | Faithful replay |
|---|---|---|
| Learned model | 18.6%3rd | 17.2%1st |
| Hybrid: failures + change | 10.3%2nd | 19.4%2nd |
| Recent failures first | 8.9%1st | 28.3%3rd |
| Change match, cheapest first | 31.2%4th | 34.8%4th |
Judged by the full replay, the learned model comes third, well behind recent failures first. Judged faithfully, it comes first: it lets 17.2% of failing builds escape, while recent failures first falls to third with 28.3%. A team choosing from the usual replay would pick recent failures first, which in production does worse than both the learned model and the hybrid.
Making selection safe and self-correcting
Knowing the problem, can we still run selection safely? The challenge asks for exactly this: bound the risk of skipping a failing test, and stay accurate as the code base changes without constant recalibration. The design I tested has two parts.
- Random audits. On a random 10% of builds, the full suite runs anyway. Audits show what the selection would have missed, without the bias of only seeing what it chose to run.
- An online controller. After every audited failure it adjusts the budget: up when the selection missed it, down when it caught it. Over time this steers the escape rate toward a target, here 10%. From what it can see, the failures it catches and the misses the audits reveal, a deployment can also compute a statistical certificate: an interval for its own escape rate.
- Points off a 10% target: online vs a budget tuned once
- 0.8 vs 4.0
- Runs where the certificate missed the true rate
- 0 of 180
- Test time saved after audits and selector time
- 28–63%
- Fewer escapes when the full suite runs every 8th build
- 14.8 pts
On the held-out builds, the online controller stayed on average 0.8 points from its 10% target, while a budget tuned once on older builds drifted 4.0 points away. On the newer builds it was 2.0 points off, against 4.9 for the tuned budget. The certificate never missed: in 30 seeds on each of six projects, the interval contained the true escape rate every time. And even after paying for the audits and for its own computation, the controller saved 28% to 63% of test time on Hive, James and Kafka (28% to 66% on the newer builds).
Scheduled full runs, the classic “we run everything nightly” fallback, also help. Running the full suite on every 8th build gave the history order enough fresh results to cut its escapes by 14.8 points at the same selection budget. In my design, though, only the random audits feed the certificate.
Is it worth it?
The challenge also asks whether selection pays for itself, in money and in energy. On the newer builds, recent failures first under the controller saves about 8,400 test-hours per 1,000 builds, after audits and selector time. At GitHub's price for hosted Linux runners, that is roughly $3,000 to $12,000 per 1,000 builds, depending on machine size.
Censoring takes a large bite out of that. To protect as well as it appears to under a full replay, recent failures first needs about 3,900 more test-hours per 1,000 builds, almost half as much as the controller saves. Orders that read the change need far less.
| Order | Extra test-hours per 1,000 builds |
|---|---|
| Recent failures first | 3,936 h |
| Hybrid: failures + change | 1,304 h |
| Change match, cheapest first | 280 h |
01k2k3k4k |
What reading the change buys
If history pays for censoring and the change does not, why not simply read the change? Because the answer depends on which failures you promise to catch.
Certified on all failures
| Order | Share of test time |
|---|---|
| Recent failures first | 38.5% |
| Hybrid: failures + change | 39.2% |
| Change match, cheapest first | 61.2% |
0%35%70% |
Certified on new failures
| Order | Share of test time |
|---|---|
| Hybrid: failures + change | 40.4% |
| Change match, cheapest first | 48.5% |
| Recent failures first | 50.0% |
0%35%70% |
Certified on all failures, recent failures first is the cheapest way to hold the target. Many failing builds are repeats of something that already broke, and history is very good at catching repeats. But the failures that matter most are often new ones, tests the pipeline has never seen fail. Certified on those, history becomes the most expensive order (50.0% of test time), and the hybrid that fuses history with the change needs about 10 points less.
Reading the change also beats the structural footprint. Ordering tests by how far their module sits from the changed modules in the build's dependency graph needed 7.7 points more test time than the change match for the same risk.
And what about LLMs? On 232 held-out builds where a test failed for the first time, I asked Gemini 3.5 Flash-Lite to read each change and rank the tests by risk.
| Order | Share of builds |
|---|---|
| Gemini 3.5 Flash-Lite | 84.1% |
| Change match, cheapest first | 81.9% |
| Shortest first | 72.4% |
| Recent failures first | 33.6% |
0%25%50%75%100% |
The LLM put the first-time failure within the first 5% of test time in 84.1% of these builds, against 81.9% for the change match. That difference of 2.2 points is too small to rule out chance (the 95% interval runs from −0.5 to +5.0 points). Both read the change, and both are far ahead of recent failures first, which finds only 33.6%: a test failing for the first time has no failure history to go on. Most of the value comes from reading the change at all, and a cheap text match already captures most of it.
Takeaways
- Evaluate selectors faithfully. Hide the results of tests a selector skipped and of builds that had not finished. Without that, history-based selectors look far better than they are, and the ranking between selectors can flip.
- Audit at random and let a controller steer. A small random sample of full runs keeps the risk measurable, and an online controller holds the target as the project changes.
- Decide which failures you are protecting against. If repeats are fine to catch late and new failures are not, read the change.
- Start cheap. Plain text matching between the change and the tests gets most of the benefit of reading the change. An LLM adds a little at most.
References
- 1. Runxiang Cheng, Shuai Wang, Reyhaneh Jabbarvand and Darko Marinov (2024). Revisiting Test-Case Prioritization on Long-Running Test Suites. ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 615–627.
- 2. Helge Spieker, Arnaud Gotlieb, Dusica Marijan and Morten Mossige (2017). Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration. ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA).
- 3. Mateusz Machalica, Alex Samylkin, Meredith Porth and Satish Chandra (2019). Predictive Test Selection. International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).