Flaky Tests & Reliability
Flaky test
Also known as: flaky test, test flakiness, intermittent test failure
A flaky test produces both passing and failing results with unchanged application and test code under comparable conditions. That intermittency can come from the test, the application, or its environment; it does not rule out a real bug.
Possible causes include timing and race conditions, missing isolation, shared state, remote services, test order, and resource leaks. Some are problems in the test; others are defects in the application or changes in its environment. The same commit does not guarantee the same inputs, configuration, dependencies, or execution conditions.
A passing retry is evidence of intermittency, not proof that the original failure was harmless or that its cause is fixed. Keep logs and artifacts from both outcomes, then compare the conditions and investigate. Quarantine can reduce disruption while that work proceeds, but it can also hide real failures.
In 2016, Google reported that almost 16% of the tests in its own corpus had some level of flakiness. That historical observation is not a current industry rate. Repeated unexplained failures can erode confidence in a suite and make genuine regressions easier to overlook.
- Unchanged code with mixed outcomes calls for investigation, not a “not a real bug” verdict.
- Compare inputs, configuration, test order or random seed, dependencies, and environment alongside the commit.
- Preserve failed-run evidence even when a retry passes; rerunning is not the same as fixing the cause.
Frequently asked
How do you know a test is flaky and not just failing?
Compare the test’s outcome history on unchanged application and test code, checking that inputs, configuration, test order, dependencies, and environment are comparable. Mixed outcomes show intermittent behavior, but do not identify its cause or rule out an application defect. Keep evidence from both failed and passing runs; no fixed number of successful retries proves that the underlying problem is gone.
Related terms
Sources
See it in your own test results
Qualflare detects flaky tests, clusters failures by root cause, and scores release risk from the test results you already produce in CI. Start free.
Start free with Qualflare← Back to the testing & observability glossary.
Last reviewed November 15, 2026