Flaky test detection
Flaky test detection uses retry evidence and result history to find tests that pass and fail on unchanged code. Those signals help you investigate intermittent behavior and prioritize repairs — they do not identify the cause or prove a release is safe.
Why flaky tests are worth detecting
A flaky test produces different results on unchanged code — usually from non-determinism: timing and race conditions, shared or external state, network calls, animations, or test-ordering dependencies. An intermittent failure can expose a real application bug, including a concurrency defect; a passing retry does not rule out a regression. Repeated uncertainty also erodes trust: teams may start dismissing red builds that need investigation. In May 2016, Google reported that almost 16% of its tests had some level of flakiness — a historical observation about Google's tests, not industry-wide prevalence. See flaky test statistics.
of Google's tests had some level of flakiness, as reported in May 2016
Non-deterministic tests have two problems, firstly they are useless, secondly they are a virulent infection that can completely ruin your entire test suite.
How flaky test detection works
One execution outcome cannot demonstrate variability. A single CI run containing a failing attempt and a passing retry can. Compare the code, inputs, configuration and execution conditions: the same commit alone does not make every other condition identical. History across runs helps reveal recurrence and context, but a history-based flag is a lead to investigate, not a diagnosis.
- Collect evidence. Keep final outcomes and available retry details, with branch, commit and environment context — see flaky test detection.
- Define the metric. Distinguish a retry-based flaky-run count, a historical case score and a failure proportion. State the observation window and denominator when reporting a flake rate.
- Assess temporary quarantine. If you remove a test from the blocking path, keep it running visibly, assign an owner and review deadline, and account for lost release-gating coverage. Use your runner or CI configuration — see test quarantine.
- Repair and re-evaluate. Investigate the test, application and environment. Define criteria for restoring blocking status. A falling failure rate or clean streak supports improvement; it does not prove the cause is gone.
What a flaky test looks like in practice
Suppose a checkout test runs once initially in each of 200 CI builds: 184 initial attempts pass and 16 fail. Each failed attempt later passes on a retry of the same commit. The initial-attempt failure proportion is 16 / 200 = 8%; the later retries are not part of that denominator. Fail-then-pass outcomes show intermittent behavior, not that the application code is innocent. This example is not Qualflare's historical case score or its dashboard Flake Rate.
Now suppose 12 of those failures timed out waiting for a network response on busy afternoon runners. Investigate runner load, response latency, dependencies, timeout assumptions and synchronization; the timeout pattern does not establish a race condition or exclude a regression. Compare how many executions used those runners before inferring elevated risk. Whether to quarantine depends on coverage, failure consequences and repair capacity — there is no universal 8% cutoff.
Detecting flaky tests by framework
Enabling retries in a runner is not enough: the reporter must retain evidence that the importer understands. The following distinctions apply to Qualflare CLI 0.5.0's standard report parsers. See the supported formats and reporter setup links for the current collection paths.
- Playwright JSON — with retries enabled, the JSON report contains attempt results. Qualflare reads a retry count and marks a passed-after-retry result flaky, but this parser does not retain complete earlier-attempt details. See Playwright flaky tests in CI.
- Cypress JUnit — ordinary JUnit output is not a guarantee that retry evidence survives collection. Qualflare's separate Cypress JSON parser reads
attemptsor_retrieswhen the report supplies them; check your reporter's actual output rather than assuming every Mochawesome configuration includes them. See Cypress flaky tests. - Jest JSON —
jest.retryTimes(n)enables runner retries, but Qualflare's standard Jest JSON parser reads final outcomes, not retry counts or attempt history. A final green result can therefore hide earlier failures. See Jest flaky tests in CI. - pytest rerun XML — when
pytest-rerunfailuresproduces adjacent entries for attempts of the same test, Qualflare's Python parser combines them, counts retries and flags a passing final result. Use the Python format; generic JUnit parsing is not equivalent. Verify your plugin version and report shape. See pytest flaky tests. - Other JUnit XML — supported XML can carry final results without usable retry detail. The generic parser reads retry counts from
retriesorretryCountproperties; it does not interpretflakyFailureorrerunFailureextensions.
Retry counts, flaky flags and full attempt histories are different data. Check the parsed report against the actual test run before uploading. History can reveal changing final outcomes, but it cannot reconstruct retry failures discarded from every final green report.
Flaky test detection with Qualflare
Qualflare distinguishes flaky runs from flaky cases. A flaky run passed after a failed attempt, where supported retry evidence is available. A flaky case is flagged from historical outcomes: by default, at least 10 qualifying results in the last 25 launches, with both passes and failures, and the less common outcome making up at least 30%.
The case score is that minority share, capped at 50%; errors, timeouts and aborted results count as failures, while skipped and pending results are ignored. The calculation uses one qualifying result per launch and does not restrict the history to a single commit. It is retrospective, not a prediction of the next failure. The dashboard's Flake Rate counts flaky runs instead. Read the detection rules and project settings.
You can keep your existing tests, but need a Qualflare project, an authenticated CLI project alias and fresh supported report files. Configure retries and a suitable reporter when you need retry-based signals. Follow the setup guide and current collection instructions: inspect a dry run, collect failed runs too, and do not make collection conditional on tests passing. Statistical flaky detection is available on Starter and Open Source as well as paid plans, and works with project AI off. Qualflare does not quarantine, skip or re-run tests.
Adjacent AI features have separate conditions. Launch analysis runs asynchronously for uploaded launches with project AI enabled and the applicable allowance; its risk label comes from observed pass rate, not a prediction that a release is safe. Failure clustering also requires AI and plan allowance, and a cluster needs five failures before it is named. Starter includes 25 monthly report analyses and holds five clusters; Open Source does not include clustering. The AI Insights page requires Core or above, unlike statistical flaky detection itself.
For investigation techniques, read the guide to flaky tests and how to detect flaky tests.
Framework-specific fixes: Cypress, Playwright, Jest, and pytest.
Not sure it's worth the effort? Put your own numbers into the flaky test cost calculator to see what these failures cost your team in triage time each month.
Detect flaky tests in your CI
Start free — collect supported results, build a history and investigate flagged tests.
Get Started FreeFrequently asked questions
What is flaky test detection?
Flaky test detection identifies tests whose outcomes vary on unchanged code. A failed attempt followed by a passing retry can reveal intermittent behavior within one CI run; history across runs helps measure recurrence and context. Neither signal alone identifies the cause or proves the application is safe.
How do you detect a flaky test?
Compare attempts on the same code and check whether inputs, configuration, dependencies or execution conditions changed. Preserve retry evidence in a supported report and collect results across CI runs. Qualflare distinguishes retry-based flaky runs from cases flagged by historical pass/fail patterns; a case score is not proof of unchanged-code nondeterminism or a prediction of release safety.
Why are flaky tests a problem?
Flaky tests consume investigation time and erode trust in red builds. They can also expose real application bugs, so dismissing them can hide regressions. In May 2016, Google reported that almost 16% of its tests had some level of flakiness; that is a historical observation about Google’s tests, not current industry prevalence.
How do you fix flaky tests once detected?
Investigate the test, application and execution environment, then repair the cause. If temporary quarantine is appropriate, use your runner or CI configuration, keep the test running visibly, assign an owner and review deadline, and account for lost blocking coverage. Define when it can return to the blocking suite. Qualflare does not quarantine, skip or re-run tests; improving results support a fix but do not prove the cause is eliminated.
Related: test observability · test reporting guides · testing & observability glossary · compare Qualflare to other tools
Sources
- Google Testing Blog — Flaky Tests at Google
- Martin Fowler — Eradicating Non-Determinism in Tests
- Luo, Hariri, Eloussi, Marinov — An Empirical Analysis of Flaky Tests (FSE 2014)
- Qualflare docs — Detect and fix flaky tests
- Qualflare docs — Supported report formats
- Qualflare docs — AI Launch Analysis
- Qualflare docs — Failure Clusters
Recent updates
Written by İbrahim Süren, founder of Qualflare. Last reviewed November 17, 2026.