How to Root-Cause a Flaky Test (2026)

Root-causing a flaky test differs from knowing why flaky tests happen in general: you already have one test failing intermittently, and the question isn’t “what are the ten categories of flakiness” but “which one is this, and how do I prove it.” The workflow is five steps — confirm a real flake rate by rerunning the test in isolation, read its failure signature, correlate when it started flaking against deploys and CI changes, use framework flags to test each hypothesis, and check whether AI failure clustering groups it with other failures. Where we reference Qualflare, our own platform, we describe only what it actually does.

For the taxonomy of why tests flake — timing, shared state, ordering, external dependencies, concurrency, environment — see the complete guide to flaky tests and flaky test statistics, which covers Luo et al.’s ten-category breakdown with percentages. This post assumes one red test in front of you, and a process for diagnosing it rather than a list of causes.

How do you confirm a test is actually flaky before you investigate it?

Don’t start reading code on the first red run — a single failure could be a real bug just as easily as a flake. Flakiness only reveals itself across history, and for one suspected test the fastest way to get that history is bisection by repetition: run just that test, in isolation, N times, and calculate its actual flake rate.

How many runs is enough depends on how rare the flake is: 20-30 isolated reruns catch most; rarer ones need more. Microsoft’s research team reran suspect tests 100 times locally, specifically to gather enough passing-and-failing logs to diff against each other (Lam et al., ISSTA 2019). The caveat from that same study is the reason this step can’t be skipped: when Microsoft reran 100 known-flaky tests locally, 86% of them never failed once outside the CI pipeline (Lam et al., ISSTA 2019). A test that’s rock-solid across 100 local runs but still flakes in CI hasn’t ruled itself out as flaky — it’s ruled out a purely code-level bug and pointed the investigation at CI-specific causes instead: parallelism, resource contention, or environment differences, exactly where the next step goes. Every major framework has a native way to run this bisection without touching the rest of the suite:

# Playwright: run one spec 30 times in isolation
npx playwright test tests/checkout.spec.ts --repeat-each=30 --workers=1

# pytest: rerun one test N times (requires the pytest-repeat plugin)
pytest tests/test_checkout.py::test_apply_coupon --count=30 -p no:randomly

# Jest: loop a single test file N times, serially
for i in {1..30}; do npx jest checkout.test.js --runInBand || echo "failed run $i"; done

With a real flake rate in hand — and CI-specific causes now on the table if local runs stayed clean — the next step is narrowing which category of cause you’re actually looking at.

How do you read a flaky test’s failure log for its root cause?

Once you have a real flake rate, the failure log usually hints at the cause category before you’ve written a line of debug code — the error message and its shape are diagnostic clues, not just noise to page past. A TimeoutError or an assertion that never becomes true almost always traces to async/timing: something the test waited on didn’t finish before the check ran. A “no such element” or StaleElementReferenceException in browser automation usually means a race condition — the UI mutated mid-interaction, out from under the test. An unexpected row count or a unique-key violation points at shared-state contamination between tests, usually a fixture that didn’t tear down cleanly, while ECONNRESET or a 5xx from one specific host points outward, at an external dependency rather than the test itself. This is about matching the error in front of you to the taxonomy already covered in flaky test statistics — not re-deriving it, just applying it fast. The full signature-to-cause mapping covers six patterns:

What the failure looks like	Likely cause	Where to look next
TimeoutError, “exceeded timeout of Xms”, an assertion that never becomes true	Async wait / timing	The awaited condition, animation, or network call that never finished
“no such element”, StaleElementReferenceException, element detached from DOM	Race condition in browser automation	The UI mutated mid-interaction; a missing web-first wait
Unique/duplicate-key violation, unexpected row count, “expected 3 got 4”	Shared-state contamination	Fixture teardown, database isolation between tests
ECONNRESET, connection refused, 5xx from one specific host	External dependency	Third-party API or service instability
Passes alone, fails in the full suite; different result depending on run order	Test-order dependency	State a preceding test leaves behind
Only fails with parallel workers, or “port already in use”	Concurrency / resource contention	A shared port, file, or global fixture across workers

None of these signatures is proof by itself — a timeout can also mean a real regression made an endpoint slower — but each narrows where to look first. Pairing the signature with the isolation-flag tests below turns a guess into a confirmed diagnosis.

How do you correlate a flaky test with deploys or CI infrastructure changes?

A test that’s been stable for months and suddenly starts flaking rarely has an explanation inside the test itself — something around it changed, and the fix is finding what. Three categories of change are worth checking against the date flakiness started: a deploy or merge to the code the test exercises, which may have introduced a genuine race condition; a CI configuration change, such as more parallel jobs, a new shard count, a bumped runner image, or smaller shared-tenancy machines; or an infrastructure change, like a database or queue that moved tiers, or a new proxy sitting in the test environment. Which category it falls into changes where you spend your time — a test that started flaking the same day CI parallelism jumped from 4 workers to 16 is almost certainly shared-port or resource contention, so the fix is isolation or worker configuration, not the business logic under test. One that started flaking right after a merge is more likely a race condition that merge introduced, the same mechanism behind why tests so often pass locally but fail in CI. Chromium’s V8 project runs this same correlation at infrastructure scale: its flake-bisect tooling calibrates the reruns needed to reliably reproduce a flake, then bisects backward through revisions to find the first one where the flake rate changed (V8 — Flake bisect). Check what changed on or just before the date it started flaking:

• A deploy or merge to the code the test exercises — may have introduced a genuine race condition.
• A CI configuration change — more parallel jobs, a new shard count, a bumped runner image, or smaller/shared-tenancy machines.
• An infrastructure change — a database or queue that changed tiers, or a new proxy in the test environment.

The manual version of V8’s bisect tooling is git log against CI build history: find the earliest run where the test starts flaking, and diff everything that changed around it — code, CI config, and infrastructure alike.

Which framework flags help you isolate the cause fastest?

Each flag tests exactly one variable at a time, holding the repeat count from the bisection step constant so you’re comparing flake rate to flake rate, not one lucky (or unlucky) run to another. The three isolate different suspects: Playwright’s trace viewer doesn’t rule anything in or out directly — it lets you see the failure, DOM snapshots and network calls and console logs scrubbed frame-by-frame at the moment things went wrong, turning a stack-trace guess into direct evidence. pytest’s -p no:randomly flag rules test-order dependency in or out by disabling randomization for one run; if the flake rate drops to zero with ordering fixed, you’ve confirmed the cause and can bisect the preceding tests to find which one leaves state behind. Jest’s --runInBand does the same job for worker parallelism, running every test file serially in the main process instead of Jest’s worker pool — if the flake disappears serially but persists under default parallel workers, the cause is cross-worker resource contention, not the test’s own logic. Run the flag that matches your current hypothesis before touching a single line of the test itself; a fix aimed at the wrong variable won’t hold.

Playwright — capture and replay the failing run. Capture traces only on runs that need them, then open the failing one:

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: { trace: 'on-first-retry' },
});

npx playwright show-trace test-results/checkout-should-apply-coupon/trace.zip

For a trace pulled from a CI artifact, drop it into the hosted Playwright Trace Viewer — no install needed. The trace viewer scrubs frame-by-frame through DOM snapshots, network calls, and console logs at the moment of failure — the fastest way to see a race condition rather than infer one from a stack trace.

pytest — rule test-order dependency in or out. If your suite uses pytest-randomly, -p no:randomly disables it for one run, giving a deterministic baseline:

pytest -p no:randomly tests/test_checkout.py

Run the suspect test with randomization off and on, keeping the repeat count constant. If the flake rate drops to zero with -p no:randomly, you’ve confirmed a test-order dependency — bisect the preceding tests one at a time to find which leaves the state behind.

Jest — rule out worker parallelism. --runInBand runs every test file serially in the main process instead of Jest’s worker pool:

npx jest checkout.test.js --runInBand

If the flake rate goes to zero with --runInBand but stays nonzero under default parallel workers, the cause is cross-worker resource contention, not the test’s own logic. See the deeper guides for Playwright, pytest, and Jest & Vitest.

Does AI failure clustering help root-cause a single flaky test?

Failure clustering is usually a whole-run triage tool: it collapses hundreds of failures into a handful of root causes across a launch. It’s also a useful diagnostic input for one test — check which cluster your suspect test’s failure lands in before spending an hour reading its code.

If clustering groups this failure with a dozen other tests sharing the same stack trace, error signature, or timing window, that’s a strong signal the cause is systemic: a broken shared fixture, an unstable dependency, or an infrastructure blip touching many tests at once — not something specific to this test’s logic. If the failure sits alone in its own cluster, the cause is more likely local, such as a race condition or a missed wait belonging to that test alone. This reuses the same non-determinism signal clustering already uses for flaky-vs-real triage, pointed at one test instead of a whole run.

Qualflare’s flaky test detection runs this clustering per launch and labels each cluster with its likely cause, so a test on your desk gets its cluster membership checked in seconds rather than guessed at.

The diagnostic workflow, at a glance

Five steps take you from a suspicious red build to a confirmed diagnosis, in order: bisect by rerunning the suspect test in isolation to get a real flake rate, read its failure signature to narrow the likely cause category, correlate when it started against deploy and CI-infrastructure history, isolate the hypothesis with the framework flag that tests it directly, and check whether AI failure clustering groups the failure with others or leaves it standing alone. Each step answers a different question and rules out a different set of causes — skipping straight to a fix without working through them is how a real, reproducible bug gets misdiagnosed as “just flaky.” The workflow is worth running as a matter of course, not just for stubborn cases, given how common flakiness is at scale (see flaky test statistics) — most teams are running some version of this process every week, whether they’ve named it or not.

Step	Question it answers	Technique
1. Bisect	Is it actually flaky, and how often?	Repeat-run flags (--repeat-each, --count, a loop with --runInBand)
2. Read the signature	Which cause category is likely?	Error/stack-trace pattern matching
3. Correlate	Did this start with a deploy or an infra change?	Git history vs. CI config history
4. Isolate	Confirm the hypothesis	Trace viewer, -p no:randomly, --runInBand
5. Check the cluster	Is it systemic or test-specific?	AI failure clustering

Once diagnosed, quarantine the test while you fix it, so it stops blocking releases without going invisible.

Start free with Qualflare — upload a CI run and see whether your suspect test’s failure clusters with others or stands alone, before you spend an hour reading its code.

Frequently asked questions

How many times should you rerun a test to confirm it’s flaky?

Enough to see the failure repeat under identical conditions — 20 to 30 isolated reruns catch most flakes, and rarer ones may need up to 100, the count Microsoft Research used to gather enough passing-and-failing logs to diff (Lam et al., ISSTA 2019). Use a framework repeat flag, such as Playwright’s —repeat-each or pytest-repeat’s —count, rather than rerunning the whole suite.

Can a CI parallelism change cause a test to start flaking?

Yes. If a test was stable for months and started flaking the same day your CI runner count, shard count, or worker parallelism changed, the likely cause is resource contention — a shared port, file, or fixture that only collides once enough tests run concurrently — not a bug in the test’s own logic.

How does Playwright’s trace viewer help debug a flaky test?

Configuring trace: ‘on-first-retry’ captures a trace only on runs that fail and retry. Opening it with npx playwright show-trace, or dropping a CI artifact into trace.playwright.dev, lets you scrub frame-by-frame through DOM snapshots, network calls, and console logs at the moment of failure — direct evidence instead of a stack-trace guess.

Does AI failure clustering help diagnose a single flaky test?

Yes — check which cluster the failure lands in before debugging it alone. Grouped with many other tests sharing a stack trace or timing window means the cause is systemic (a shared fixture, dependency, or infrastructure blip), and fixing it clears the whole cluster. Alone in its own cluster means the cause is more likely local to that test.