Flaky test detection

Flaky test detection finds the tests that pass and fail on the same code — the false signals that quietly destroy trust in your suite — by scoring each test's pass/fail history across runs, so you can quarantine and fix them before they block a release.

Why flaky tests are worth detecting

A flaky test produces different results on unchanged code — usually from non-determinism: timing and race conditions, shared or external state, network calls, animations, or test-ordering dependencies. Because the code under test did not change, a flaky failure tells you nothing reliable. Its real cost is trust: once a suite cries wolf often enough, teams start ignoring red builds — and a genuine regression slips through. Google has reported that roughly 16% of its tests exhibit some flakiness; see the data in flaky test statistics.

How flaky test detection works

You cannot tell a test is flaky from a single run — flakiness is a property of behavior over time. Detection therefore works from history: collect results from every CI run, track each test's outcome across them, and flag the tests whose results vary on unchanged code. Test-runner retries are the strongest raw signal — a test that fails and then passes on retry is flaky by definition. From enough history you can go further and assign a predictive flaky score, flagging unreliable tests before they disrupt a deploy rather than after.

• Collect history. Upload each CI run's results so every test has an outcome record across runs and branches — see flaky test detection.
• Score flakiness. Use retry signals and pass/fail history to compute a flake rate per test, suite, and pipeline.
• Quarantine. Move known-flaky tests out of the blocking path so they stop failing builds while you fix them — see test quarantine.
• Fix the root cause, then verify. Track flake rate over time to confirm the fix worked instead of muting the symptom.

What a flaky test looks like in practice

Take one end-to-end test — a checkout flow — that ran on 200 CI builds this month. It passed 184 times and failed 16, and every one of those failures was on a commit that passed on a later run of the same build. That is an 8% flake rate, and it tells you the code was never the problem. Look closer and the pattern sharpens: 12 of the 16 failures were the same assertion timing out waiting on a network response, and all 12 landed on the busiest runners in the afternoon. That is the signature of a race condition, not a regression.

This is why a single result can't classify a test, and why a human eyeballing one red build can't either. The flake only exists in the distribution of outcomes — you need the history to see it, and once you can see it you can act: the retry that passed confirms the test is flaky, the 8% rate says it is bad enough to quarantine, and the shared timeout signature says where to look for the fix.

Detecting flaky tests by framework

Most runners already produce a flakiness signal — a retry — and record it in their result files. Detecting flakes at scale is a matter of collecting those signals across every run rather than reading one report. Point your runner at a results file, then hand it to the Qualflare CLI:

• Playwright — set retries in playwright.config.ts; a test that fails then passes is marked flaky. Emit the JSON reporter, then qf my-project collect playwright-results.json. See Playwright flaky tests in CI.
• Cypress — enable test retries (retries in cypress.config); each attempt is recorded. Emit JUnit, then qf my-project collect cypress-results.xml.
• Jest — jest.retryTimes(n) re-runs failed tests; write jest --json --outputFile=jest-results.json, then qf my-project collect jest-results.json. See Jest flaky tests in CI.
• pytest — pytest-rerunfailures (--reruns n) marks reruns; emit pytest --junitxml=results.xml, then qf my-project collect results.xml. See pytest flaky tests.
• Anything else — any framework that emits JUnit XML works: qf my-project collect junit.xml.

One caveat: retries hide flakiness as much as they reveal it. A test that fails then passes turns the build green, so the flake never appears in a pass/fail report — only in the retry metadata. That is exactly why detection has to score history across runs instead of trusting a single green.

Flaky test detection with Qualflare

Qualflare detects flaky tests from the results your CI already produces — no change to how your tests are written. It captures retry counts and flaky status automatically, scores flakiness across runs, and surfaces the worst offenders alongside failure clustering and per-launch risk. Follow the five-step, framework-agnostic setup guide, or start from the complete guide to flaky tests and how to detect flaky tests.

Framework-specific fixes: Cypress, Playwright, Jest, and pytest.

Not sure it's worth the effort? Put your own numbers into the flaky test cost calculator to see what these failures cost your team in triage time each month.

Frequently asked questions

Related: test observability · test reporting guides · testing & observability glossary · compare Qualflare to other tools