How to Root-Cause a Flaky Test: A Step-by-Step Diagnostic Workflow (2026)

Root-causing a flaky test differs from knowing why flaky tests happen in general: you already have one test failing intermittently, and the question isn’t “what are the ten categories of flakiness” but “which one is this, and how do I prove it.” The workflow is five steps — confirm a real flake rate by rerunning the test in isolation, read its failure signature, correlate when it started flaking against deploys and CI changes, use framework flags to test each hypothesis, and check whether AI failure clustering groups it with other failures. Where we reference Qualflare, our own platform, we describe only what it actually does.

For the taxonomy of why tests flake — timing, shared state, ordering, external dependencies, concurrency, environment — see the complete guide to flaky tests and flaky test statistics, which covers Luo et al.’s ten-category breakdown with percentages. This post assumes one red test in front of you, and a process for diagnosing it rather than a list of causes.

How do you confirm a test is actually flaky before you investigate it?

Don’t start reading code on the first red run — a single failure could be a real bug just as easily as a flake. Flakiness only reveals itself across history, and for one suspected test the fastest way to get that history is bisection by repetition: run just that test, in isolation, N times, and calculate its actual flake rate.

How many runs is enough depends on how rare the flake is: 20-30 isolated reruns catch most; rarer ones need more. Microsoft’s research team reran suspect tests 100 times locally, specifically to gather enough passing-and-failing logs to diff against each other (Lam et al., ISSTA 2019). Every major framework has a native way to do this without touching the rest of the suite:

# Playwright: run one spec 30 times in isolation
npx playwright test tests/checkout.spec.ts --repeat-each=30 --workers=1

# pytest: rerun one test N times (requires the pytest-repeat plugin)
pytest tests/test_checkout.py::test_apply_coupon --count=30 -p no:randomly

# Jest: loop a single test file N times, serially
for i in {1..30}; do npx jest checkout.test.js --runInBand || echo "failed run $i"; done

The caveat comes from that same study: when Microsoft reran 100 known-flaky tests locally, 86% of them never failed once outside the CI pipeline (Lam et al., ISSTA 2019). A suspect test that’s rock-solid across 100 local runs but still flakes in CI hasn’t ruled itself out — it’s ruled out a purely code-level bug and pointed you at CI-specific causes: parallelism, resource contention, or environment differences, exactly where the next step goes.

How do you read a flaky test’s failure log for its root cause?

Once you have a real flake rate, the failure log usually hints at the cause category before you’ve written a line of debug code. This is about matching the error in front of you to the taxonomy already covered in flaky test statistics — not re-deriving it, just applying it fast:

What the failure looks like	Likely cause	Where to look next
TimeoutError, “exceeded timeout of Xms”, an assertion that never becomes true	Async wait / timing	The awaited condition, animation, or network call that never finished
”no such element”, StaleElementReferenceException, element detached from DOM	Race condition in browser automation	The UI mutated mid-interaction; a missing web-first wait
Unique/duplicate-key violation, unexpected row count, “expected 3 got 4”	Shared-state contamination	Fixture teardown, database isolation between tests
ECONNRESET, connection refused, 5xx from one specific host	External dependency	Third-party API or service instability
Passes alone, fails in the full suite; different result depending on run order	Test-order dependency	State a preceding test leaves behind
Only fails with parallel workers, or “port already in use”	Concurrency / resource contention	A shared port, file, or global fixture across workers

None of these signatures is proof by itself — a timeout can also mean a real regression made an endpoint slower — but each narrows where to look first. Pairing the signature with the isolation-flag tests below turns a guess into a confirmed diagnosis.

How do you correlate a flaky test with deploys or CI infrastructure changes?

A test that’s been stable for months and suddenly starts flaking rarely has an explanation inside the test itself — something around it changed. Check what changed on or just before the date it started flaking:

• A deploy or merge to the code the test exercises — may have introduced a genuine race condition.
• A CI configuration change — more parallel jobs, a new shard count, a bumped runner image, or smaller/shared-tenancy machines.
• An infrastructure change — a database or queue that changed tiers, or a new proxy in the test environment.

The distinction changes where you spend your time. A test that started flaking the same day CI parallelism went from 4 workers to 16 is likely shared-port or resource contention — fix isolation or worker configuration, not the business logic under test. One that started flaking right after a merge is likely a race condition that merge introduced, which is also why tests so often pass locally but fail in CI.

Chromium’s V8 project applies the same logic at the infrastructure level: its flake-bisect tooling calibrates reruns needed to reliably reproduce a flake, then bisects backward through revisions to find the first one where the flake rate changed (V8 — Flake bisect). Apply the same idea manually with git log and CI build history: find the earliest run where the test starts flaking, and diff what changed around it — code, CI config, and infrastructure alike.

Which framework flags help you isolate the cause fastest?

Each flag tests exactly one variable, holding the repeat count from step one constant so you compare flake rate to flake rate, not one run to one run.

Playwright — capture and replay the failing run. Capture traces only on runs that need them, then open the failing one:

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: { trace: 'on-first-retry' },
});

npx playwright show-trace test-results/checkout-should-apply-coupon/trace.zip

For a trace pulled from a CI artifact, drop it into the hosted Playwright Trace Viewer — no install needed. The trace viewer scrubs frame-by-frame through DOM snapshots, network calls, and console logs at the moment of failure — the fastest way to see a race condition rather than infer one from a stack trace.

pytest — rule test-order dependency in or out. If your suite uses pytest-randomly, -p no:randomly disables it for one run, giving a deterministic baseline:

pytest -p no:randomly tests/test_checkout.py

Run the suspect test with randomization off and on, keeping the repeat count constant. If the flake rate drops to zero with -p no:randomly, you’ve confirmed a test-order dependency — bisect the preceding tests one at a time to find which leaves the state behind.

Jest — rule out worker parallelism. --runInBand runs every test file serially in the main process instead of Jest’s worker pool:

npx jest checkout.test.js --runInBand

If the flake rate goes to zero with --runInBand but stays nonzero under default parallel workers, the cause is cross-worker resource contention, not the test’s own logic. See the deeper guides for Playwright, pytest, and Jest & Vitest.

Does AI failure clustering help root-cause a single flaky test?

Failure clustering is usually a whole-run triage tool: it collapses hundreds of failures into a handful of root causes across a launch. It’s also a useful diagnostic input for one test — check which cluster your suspect test’s failure lands in before spending an hour reading its code.

If clustering groups this failure with a dozen other tests sharing the same stack trace, error signature, or timing window, that’s a strong signal the cause is systemic: a broken shared fixture, an unstable dependency, or an infrastructure blip touching many tests at once — not something specific to this test’s logic. If the failure sits alone in its own cluster, the cause is more likely local, such as a race condition or a missed wait belonging to that test alone. This reuses the same non-determinism signal clustering already uses for flaky-vs-real triage, pointed at one test instead of a whole run.

Qualflare runs this clustering per launch and labels each cluster with its likely cause, so a test on your desk gets its cluster membership checked in seconds rather than guessed at.

The diagnostic workflow, at a glance

Step	Question it answers	Technique
1. Bisect	Is it actually flaky, and how often?	Repeat-run flags (--repeat-each, --count, a loop with --runInBand)
2. Read the signature	Which cause category is likely?	Error/stack-trace pattern matching
3. Correlate	Did this start with a deploy or an infra change?	Git history vs. CI config history
4. Isolate	Confirm the hypothesis	Trace viewer, -p no:randomly, --runInBand
5. Check the cluster	Is it systemic or test-specific?	AI failure clustering

Flakiness is common enough to warrant this process, not a one-off — Google’s own analysis found almost 16% of its tests show some flakiness (Google Testing Blog, 2016). Once diagnosed, quarantine the test while you fix it, so it stops blocking releases without going invisible.

Start free with Qualflare — upload a CI run and see whether your suspect test’s failure clusters with others or stands alone, before you spend an hour reading its code.

Frequently asked questions

How many times should you rerun a test to confirm it’s flaky?

Enough to see the failure repeat under identical conditions — 20 to 30 isolated reruns catch most flakes, and rarer ones may need up to 100, the count Microsoft Research used to gather enough passing-and-failing logs to diff (Lam et al., ISSTA 2019). Use a framework repeat flag, such as Playwright’s —repeat-each or pytest-repeat’s —count, rather than rerunning the whole suite.

Can a CI parallelism change cause a test to start flaking?

Yes. If a test was stable for months and started flaking the same day your CI runner count, shard count, or worker parallelism changed, the likely cause is resource contention — a shared port, file, or fixture that only collides once enough tests run concurrently — not a bug in the test’s own logic.

How does Playwright’s trace viewer help debug a flaky test?

Configuring trace: ‘on-first-retry’ captures a trace only on runs that fail and retry. Opening it with npx playwright show-trace, or dropping a CI artifact into trace.playwright.dev, lets you scrub frame-by-frame through DOM snapshots, network calls, and console logs at the moment of failure — direct evidence instead of a stack-trace guess.

Does AI failure clustering help diagnose a single flaky test?

Yes — check which cluster the failure lands in before debugging it alone. Grouped with many other tests sharing a stack trace or timing window means the cause is systemic (a shared fixture, dependency, or infrastructure blip), and fixing it clears the whole cluster. Alone in its own cluster means the cause is more likely local to that test.