Flaky or a Real Bug? A Triage Procedure

A test just went red. Before you can do anything useful you have to answer one question: is this a bug, or is this noise?

Most teams answer it by feel, then rerun, then move on if it goes green. That is a decision procedure — just an undocumented one with a known failure mode. What is surprising is how little of the published advice on doing it properly survives contact with its own sources.

The numbers everyone repeats do not hold up

Three thresholds circulate widely. All three are unusable, and knowing why is more useful than any of them.

“One to three failures in ten means flaky; four or more means broken.” There is no primary source. Searching for it surfaces only content-marketing sites and tool blogs citing each other. Nobody’s engineering organisation published it.

“Quarantine anything above a 5% flake rate.” Also unsourced as a published default. Datadog documents a configurable failure-rate threshold and ships no default. Cypress publishes bands — 0–10% low, 10–50% medium, above 50% high — but they colour a badge and trigger no action. No vendor and no engineering org publishes 5%.

“Compute the failure rate over the last 50 runs.” This one is real, and it comes with a twist that disqualifies it. It is from Slack’s account of handling flaky tests at scale, which describes pulling test history “for the last N test runs (for example N = 50)” and computing failed runs over N. But N = 50 is explicitly labelled an example, the real cutoff is never published — and Slack rolled the whole approach back.

Two documented reasons it failed, both of which apply to anyone building the same thing:

• New tests have no history, so the system classified them as flaky by default.
• Pull requests fixing a flaky test went falsely green, because the fix was evaluated against the test’s bad history.

Their replacement threshold is not a percentage at all. It is one: “A single failure will result in the test being disabled (regardless of reruns).”

So the most-cited heuristic in this space is a version one that its own author abandoned. That is worth knowing before you build it.

What is actually published

GitHub: make each retry answer a question

This is the best procedure in the literature, and it inverts how most teams use retries. Rather than rerunning to see if it goes green, GitHub retries three specific ways, each isolating one hypothesis:

Retry	If it now passes
Same process	“likely caused by randomness in the code or a race condition”
Same process, time shifted into the future	“incorrect assumption about time”
Different host	test order-dependence or other shared state

The third is the sharpest: if it passes on a different host while the first two still fail, the problem is not in the test — it is in what the test inherited from whatever ran before it.

The payoff is large and measured. Their previous two-method detection caught 25% of flaky failures; the three-retry approach catches 90%. Combined with impact-based ranking, flaky failures went from 9% of commits to under 0.5% — the 18× in the title.

And a finding that should change how much triage you do at all: “most flaky tests failed fewer than ten times and only 0.4 percent of flaky tests failed 100 times or more. This made one thing clear: not every flaky failure should be investigated.”

GitLab: classification as a written rule

GitLab’s handbook publishes an actual classifier with three buckets:

• Flaky — failures spread over 3+ days, still actively failing (within 3 days of the last failure)
• Master-broken — high-volume incidents: 30 or more failures in 12 hours with 40%+ concentration, or 60+ absolute
• Unclear — does not meet either

Note what distinguishes the first two: spread over time versus concentrated in time. A real breakage arrives suddenly and densely; flakiness dribbles. That is a better discriminator than any percentage because it uses the shape of the failures rather than just the count.

Their quarantine process is equally concrete: fast quarantine 3 days maximum, long-term 3 months maximum, a week’s deletion warning, automatic deletion after 3 months. Exit criteria are explicit — “The test passes consistently (more than 100 local runs), and the root cause is identified and fixed” — followed by a week of monitoring. And a rule worth stealing: “If a test is quarantined 3 or more times: The test is a candidate for permanent removal.”

They also encode the distinction in the test annotation itself, tagging a quarantined test with a type of :flaky, :bug, :stale, :broken or :investigating. The classification lives next to the code rather than in a dashboard.

Dropbox: bisect, then confirm

Dropbox’s Athena publishes the cleanest reproduce-and-confirm loop. A test is quarantined if it “flaps from passing to failing across multiple hosts.” For a suspected real breakage they “run a bisect to find a potential transition point… and rerun the test there ten times to confirm it’s truly broken and not just flaky.” On the number: “We settled on ten retries after some experimentation.”

They also state the limit honestly, which most write-ups do not: “it’s impossible to say with 100% confidence whether a commit truly broke a test.”

Rerunning is a weak classifier

This is the part that should recalibrate expectations. Reruns feel definitive and are not.

FlakeFlagger (ICSE 2021) reran full suites 10,000 times per project to establish ground truth. What that revealed about smaller rerun budgets:

Reruns	Share of flaky tests surfaced
10	~25%
100	~50%
1,000	~two-thirds

Ten reruns — far more than any CI pipeline does — finds a quarter of them. If your procedure is “rerun three times and call it flaky if it passes,” you are running a classifier with poor recall and treating its output as certainty.

DeFlaker (ICSE 2018) drew the useful conclusion: “changing the kind of rerun is more likely to help than just increasing the number of reruns.” Which is exactly what GitHub’s three-way retry does, and the strongest argument for designing retries as experiments rather than as attempts.

The signals worth checking

Assembled from what the sources above actually document:

Question	What a yes suggests
Does it fail on main too?	Real breakage, not something your branch introduced
Does it pass in the same process on retry?	Randomness or a race condition
Does it pass with the clock shifted?	A time or timezone assumption
Does it pass on a different host?	Order-dependence or shared state
Are failures dense in time, or spread over days?	Dense means broken, spread means flaky
Did it start at a specific commit?	Bisect and confirm with repeated runs at the transition
Is it a large test?	Google found binary size (r²=0.82) and RAM (r²=0.76) predict flakiness better than the tool does

That last one is counterintuitive enough to be worth stating plainly: in Google’s 2017 analysis, how big the test is predicts flakiness better than which framework it uses. Small tests were 0.5% flaky, large ones 14%.

Both errors are real, and only one is measured

Google’s 2016 post names both failure directions unusually directly.

Dismissing a real failure: “developers dismiss a failing result as flaky only to later realize that it was a legitimate failure… It is human nature to ignore alarms when there is a history of false signals.”

And over-quarantining: it “could easily mask a real race condition or some other bug in the code being tested.”

The costs of the second error — investigating noise — are measured. Slack reported roughly 28 minutes to manually triage each failure. Microsoft research put the average cost of a test-result inspection at $9.60 and found about a third were unnecessary. Google reported spending between 2% and 16% of compute resources rerunning flaky tests.

The cost of the first error — shipping a bug you called flaky — is essentially unmeasured. The closest published data point is from Eck et al. at Mozilla, where 24 of 248 failures labelled intermittent by a sheriff turned out, on investigation by the fixing developer, to be permanent failures — and even that appears as a data-cleaning note rather than a finding.

No study prospectively measures how often a failure dismissed as flaky was a real bug. No published post-mortem attributes a production incident to an ignored flake. That absence is itself worth knowing, because it means the asymmetry everyone assumes — that over-investigating is expensive and under-investigating is catastrophic — is half assumption.

What is established is the trust cost. Eck et al. found 73% of developers report a test is no longer fully reliable once it has been flaky. A 2022 study of 335 developers found trust degrades more from failures than from passes, and concluded that “considering flaky tests solemnly as false alarms is therefore a dangerous action, which cannot be recommended.”

A procedure you can actually run

1. Check main first. If it fails there too, it is not your branch. GitLab’s dense-versus-spread test applies here.
2. Retry deliberately, not repeatedly. Same process, then time-shifted, then a different host. Each answer narrows the cause.
3. If it looks real, bisect — then rerun about ten times at the transition point before believing it, as Dropbox does.
4. Rank by impact, not by incidence. Most flaky tests barely matter; a small number cost you constantly.
5. Quarantine with an expiry and an owner. Three days fast, three months maximum, deletion after that.
6. Record the classification. Flaky, bug, stale, broken — next to the test.

Two things this procedure needs that a single test run cannot give you: whether the test fails on main, and whether failures are dense or spread. Both are questions about history — which is why every published system above is built on stored results rather than on parsing a report harder.

Our own position: Qualflare is our product, and it stores CI results and scores flaky tests from history, which is the data layer steps 1 and 4 need. It does not run your tests and it does not do the retries — those belong in your CI, and GitHub’s three-way pattern is the thing to copy, free, today. The broader treatment is in the complete guide to flaky tests; the quarantine side is in flaky test quarantine strategy.

Frequently asked questions

How many times should you rerun a test to decide it is flaky?

More than you would guess, if reruns are your only tool. Research running full suites 10,000 times per project found that 10 reruns surfaces roughly 25% of flaky tests, 100 reruns about half, and 1,000 reruns around two-thirds. That is why the better approach is to vary the conditions rather than the count — changing what the rerun tests is more informative than repeating the same one.

What flake-rate threshold should trigger quarantine?

There is no published industry answer, which is worth knowing before you adopt one from a blog post. Datadog documents a configurable failure-rate threshold and ships no default. Cypress publishes severity bands but they only colour a badge and trigger no action. GitLab, whose process is the most fully documented, classifies on failure patterns — spread over three or more days and still actively failing — rather than on a percentage.

How do you tell a flaky test from a real bug?

Design each retry to answer one question. GitHub’s published approach retries three ways: in the same process, which points to randomness or a race if it passes; in the same process with time shifted forward, which points to a bad assumption about time; and on a different host, which points to order-dependence or shared state. That took their detection of flaky failures from 25% to 90%.

Should every flaky failure be investigated?

No, and GitHub’s data is the clearest argument against it. They found most flaky tests failed fewer than ten times, and only 0.4% failed 100 times or more — so they rank by impact rather than treating every flaky failure as a ticket. Triage effort should follow how much a test actually costs you, not the fact that it flaked once.

What does it cost to get this wrong?

In one direction the costs are measured: Slack reported about 28 minutes to manually triage each failure, Microsoft research put the average cost of a test-result inspection at $9.60 and found a third were unnecessary, and Google reported spending between 2% and 16% of compute resources rerunning flaky tests. In the other direction — a real bug dismissed as flaky — there is no published measurement at all.