Skip to content

Early Adopter Offer:Get 40% off Core & Scale for your first year with code EARLYQFView pricing

Flaky Tests & Reliability

Flaky test

Also known as: flaky test, test flakiness, intermittent test failure

A flaky test produces both passing and failing results with unchanged application and test code under comparable conditions. That intermittency can come from the test, the application, or its environment; it does not rule out a real bug.

Possible causes include timing and race conditions, missing isolation, shared state, remote services, test order, and resource leaks. Some are problems in the test; others are defects in the application or changes in its environment. The same commit does not guarantee the same inputs, configuration, dependencies, or execution conditions.

A passing retry is evidence of intermittency, not proof that the original failure was harmless or that its cause is fixed. Keep logs and artifacts from both outcomes, then compare the conditions and investigate. Quarantine can reduce disruption while that work proceeds, but it can also hide real failures.

In 2016, Google reported that almost 16% of the tests in its own corpus had some level of flakiness. That historical observation is not a current industry rate. Repeated unexplained failures can erode confidence in a suite and make genuine regressions easier to overlook.

Illustrative runs on unchanged code pass, fail, then pass. Possible causes branch into test code, application races or leaks, and environment or shared state. A passing retry does not prove a fix.
Illustrative outcomes, not customer data or a measured failure rate. Compare run conditions and retain the failed-run evidence: a passing retry does not establish the cause or prove that it is fixed.
  • Unchanged code with mixed outcomes calls for investigation, not a “not a real bug” verdict.
  • Compare inputs, configuration, test order or random seed, dependencies, and environment alongside the commit.
  • Preserve failed-run evidence even when a retry passes; rerunning is not the same as fixing the cause.

Frequently asked

How do you know a test is flaky and not just failing?

Compare the test’s outcome history on unchanged application and test code, checking that inputs, configuration, test order, dependencies, and environment are comparable. Mixed outcomes show intermittent behavior, but do not identify its cause or rule out an application defect. Keep evidence from both failed and passing runs; no fixed number of successful retries proves that the underlying problem is gone.

See it in your own test results

Qualflare detects flaky tests, clusters failures by root cause, and scores release risk from the test results you already produce in CI. Start free.

Start free with Qualflare

← Back to the testing & observability glossary.

Last reviewed November 15, 2026