
Product news and testing tips.
The best flaky-test tool depends on your runner, the evidence you need, and whether it must change CI gates. Retry reporting reveals failed-then-passed attempts; cross-run history shows recurring instability. Qualflare combines reporting, historical scoring and conditional AI analysis; BuildPulse and Trunk also offer quarantine; ecosystem tools suit supported integrations. Start with the reports you already generate, then compare retained attempts, history and debugging context. A flaky result can expose a real intermittent product bug.
Key takeaways
- There's no single best flaky-test tool — match it to your stack and your bottleneck, from framework-agnostic platforms to CI-native detection.
- A failed attempt followed by a passing retry is a useful flaky signal; history shows frequency, persistence and behavior after a fix.
- Mixed outcomes on unchanged code do not rule out an application bug. Investigate the cause before making a test non-blocking.
- Qualflare separates retry-based flaky runs from historical flaky cases; its case score is the minority pass/fail share, not failure rate.
- Check ingestion, plan limits and quarantine behavior. A native integration still needs configuration, and free retry software still consumes execution time.
Flaky-test debugging tools identify inconsistent test outcomes and preserve the evidence needed to investigate them. A failed attempt followed by a passing retry is an immediate signal; cross-run history reveals recurrence. Qualflare combines reporting and historical scoring with conditional AI analysis. BuildPulse and Trunk also offer quarantine, while ecosystem tools suit supported runners and CI configurations. The right choice depends on whether your bottleneck is seeing attempts, comparing history, investigating failures or changing CI gates.
This documentation-based comparison was checked November 6, 2026. Qualflare is our product. We use published documentation for capabilities and limits, not a hands-on benchmark of all eight options. Vendor descriptions establish advertised functionality, not comparative accuracy or guaranteed time savings.
The problem predates these products: Google documented its experience with flaky tests in 2016. That historical account is not a current industry prevalence estimate. Measure your own suite, and remember that intermittent behavior can originate in application code as well as the test or environment.
The 8 tools at a glance
- Qualflare — retry visibility, historical flaky scoring and conditional AI failure clustering
- BuildPulse — flaky-test detection, impact ranking and quarantine
- Trunk Flaky Tests — configurable detection and quarantine in CI workflows
- Datadog Test Optimization — flaky-test detection inside Datadog’s CI observability
- CircleCI flaky-test detection — built-in detection for teams already on CircleCI
- Cypress Cloud — flake detection and analytics for Cypress suites
- Develocity — build/test diagnostics, flaky detection and separate test-selection features
- Framework retries and retry reporting — Playwright, Jest and a pytest plugin as a starting point
How we chose these tools
A useful comparison separates five jobs:
- Detection and measurement — does it preserve retry attempts, compare stable test identities and explain its windows and thresholds?
- Integration — which local or CI reports, runners and environments does it support? Does conversion retain attempts and artifacts?
- CI action — does it only report, or can it change blocking behavior through a configured quarantine policy?
- Investigation — can you inspect failure details, traces, environment context or related failures? Grouping is a lead to investigate, not proof of one cause.
- Follow-through — can an owner review history, fix the problem and restore blocking protection where appropriate?
The entries below describe fit and limits against these criteria. No single metric substitutes for investigating an intermittent failure.
The 8 best tools for debugging flaky tests in 2026
1. Qualflare — best for history-based flaky scoring and failure clustering
Qualflare is a test management and observability platform. Its flaky test detection distinguishes a flaky run, which passed after a failed attempt, from a flaky case, classified from outcomes across launches. These are complementary forms of flaky-test detection, not interchangeable scores.
The documented defaults are a 25-launch window, at least 10 results, and a 30% minority pass/fail threshold. The case score is the less common outcome share, capped at 50%; it is not failure rate. A case failing 80% of the time has a 20% minority share and would not meet that default threshold. Manual case results count too; archived cases are excluded. The dashboard Flake Rate instead counts retry-based flaky runs.
Upload supported reports from local runs or CI with the CLI or reporters, retaining attempts where the report format supports them. AI failure clustering is a separate process that groups related failed results for investigation.
Key features:
- Retry and case history — inspect retry badges and historical case classification with configurable thresholds.
- Failure clusters — groups are named/analyzed after accumulating five failures; one small upload may not produce a named cluster.
- Launch analysis — eligible uploaded launches with project AI enabled and an active subscription can receive a pass-rate-based risk level plus AI-written explanations and recommendations.
- Reporting history — results remain until deletion; uploaded artifacts expire after 30 days, while trend charts show up to 90 days.
Best fit: teams combining reporting, historical case scoring and failure investigation across supported frameworks, with local reporting alongside CI uploads.
Limits: Qualflare does not quarantine, skip or rerun tests. Starter has 50 report uploads/month, separately 10 launch analyses/month, and two clusters held at once across the workspace. Analysis is conditional, not a release-safety verdict. Check the plan limits and current detection docs when evaluating. For broader scoring concepts, see predictive flaky scoring.
2. BuildPulse — best for detection, impact ranking and quarantine
BuildPulse offers Test Intelligence alongside CI runners. Its flaky-test product ingests JUnit XML, tracks outcomes across branches and PRs, ranks flaky tests by estimated impact, and supports manual and automatic quarantine.
The product page lists first-class GitHub Actions and Bitbucket integration, with API uploads for other CI providers. Its impact estimates help prioritize investigation; they are not independently verified savings for your team.
Key features (as described by the vendor):
- Detection and ranking — historical outcomes and estimated engineering impact.
- Quarantine — manual and automatic options to make selected failures non-blocking.
- Follow-up — Slack/email alerts and Jira/GitHub issue workflows.
Best fit: teams wanting flaky-test detection and remediation workflows from standard reports across supported frameworks.
Trade-offs: verify report identity and quarantine integration for your runner and CI. Test Intelligence is a paid product with an advertised 14-day trial; free runner minutes are a different offer. Fixing application or test code remains the responsibility of your team.
3. Trunk Flaky Tests — best for auto-quarantine in the PR workflow
Trunk ingests CI results and uses configurable monitors, including pass-on-retry, failure rate and failure count. It can quarantine detected flaky tests through a configured CI workflow, while retaining their execution history and failure details.
According to the quarantine documentation, quarantined tests continue running and reporting. The CLI can override the exit status when failures come only from quarantined tests. This removes blocking protection, not test execution; unrelated failures must still block.
Key features (as described by the vendor):
- Configurable detection — monitors classify flaky/broken behavior with configurable rules.
- Manual and automatic quarantine — changes CI gating after the required setup.
- Investigation and ownership — history, failure details, GitHub PR comments and ticketing integrations.
Best fit: teams prioritizing CI quarantine and follow-up workflows; GitHub PR comments are useful for GitHub-centric teams, while the CLI supports other CI providers.
Trade-offs: setup must preserve failures outside the quarantined set, including build errors. Assign owners and review dates so non-blocking tests receive fixes. Confirm runner/report support and current pricing rather than assuming a generic upload enables quarantine.
4. Datadog Test Optimization — best for teams already on Datadog
Datadog Test Optimization sits alongside CI Visibility within CI/CD Optimization. It detects flaky and newly flaky tests and exposes test-level history and debugging context through supported integrations.
For organizations standardized on Datadog, the appeal is connecting test signals with captured pipeline and infrastructure context — useful when tests pass locally but fail in CI. Available context depends on instrumentation and the data you collect.
Key features (as described by the vendor):
- Flaky and new-flaky detection — identifies unstable tests across runs in CI
- CI observability context — test results correlated with pipeline and infrastructure data
- Broad CI/language coverage — instrumentation across common runners and languages
Best fit: teams already invested in Datadog that want flaky-test detection in the same pane as the rest of their observability.
Trade-offs: check the specific language, runner, feature and billing requirements. Broad platform coverage does not establish equal feature support for every integration. Existing Datadog adoption can simplify evaluation, but does not remove instrumentation work.
5. CircleCI flaky-test detection — best if you already run on CircleCI
CircleCI Test Insights identifies tests that failed and passed on the same commit within 14 days. Its performance summary covers the most recent 100 runs. This is a specific detection rule, not proof that the underlying application is defect-free.
Detection needs test metadata: generate supported JUnit XML and configure the store_test_results step. The upload documentation explicitly says test metadata is not collected automatically without the required formatter/configuration.
Key features (as described by the vendor):
- Same-commit detection — mixed outcomes in the documented 14-day window.
- Test Insights — recent-run summaries, failed tests and slow tests.
- Native integration — configured result uploads keep the workflow within CircleCI.
Best fit: teams on CircleCI whose organization and pipeline type support Test Insights and whose jobs already emit suitable reports.
Trade-offs: the inspected docs support GitHub/Bitbucket OAuth organizations and exclude organizations with a circleci/<UID> slug. Check eligibility before assuming all CircleCI users receive the same insights. Reporting remains scoped to captured CircleCI executions.
6. Cypress Cloud — best for Cypress suites
Cypress Cloud provides recorded-run history, flake analytics and Test Replay for Cypress suites. Its flake management documentation requires retries and recorded runs, with flake management on Team or higher plans.
A final passing result can still be flaky if an earlier attempt failed. Cypress distinguishes failure rate from flake rate, helping teams avoid treating recovered attempts as clean first-time passes. Replay adds debugging evidence; use the complete flaky-test guide for the underlying investigation process.
Key features (as described by the vendor):
- Flake detection and analytics — flaky tests surfaced across recorded runs
- Test Replay — replay a failing run to diagnose non-determinism
- Run history — trends and results centralized for the team
Best fit: teams whose suite is Cypress-first and want flake detection plus replay tied directly to their runner. See also fixing Cypress flaky tests.
Trade-offs: Cypress-specific recording does not cover an entire polyglot suite. Verify recording limits and the plan requirements for the analytics/replay features you need; retry configuration alone does not create shared Cloud history.
7. Develocity — best for build and test diagnostics
Develocity (formerly Gradle Enterprise) combines Build Scan diagnostics, flaky detection and build/test analytics. Gradle and Maven remain strong fits, but the current flaky-detection guide also covers Bazel, sbt, selected npm runners and Python.
Within-build detection classifies failed-then-passed attempts as flaky. Cross-build detection compares outcomes with matching input fingerprints. Separately, Predictive Test Selection chooses a subset of tests from historical evidence. Unlike flaky scoring, selection changes what executes; it has coverage/speed trade-offs, not a guarantee of unchanged signal.
Key features (as described by the vendor):
- Within- and cross-build signals — retry outcomes and input-aware history, subject to integration support.
- Build Scan context — test failures alongside build diagnostics.
- Predictive Test Selection — separate selection profiles and a simulator to evaluate the trade-off.
Best fit: teams seeking build diagnostics and test analytics together, especially established Gradle/Maven estates; evaluate other agents by feature rather than assuming JVM-only support.
Trade-offs: capabilities differ by agent. The Python agent is beta and detects pytest retries through pytest-retry; flaky outcomes appear in Build Scan, not the Tests Dashboard. Do not assume every agent has identical cross-build detection or test selection. Check licensing and rollout requirements for the features you need.
8. Framework retries and retry reporting — best starting point
Start with reports from the runner you already use. Playwright labels failed-then-passed retries as flaky and can fail CI on flaky outcomes. Jest supports jest.retryTimes() and logging errors before retry with its default jest-circus runner. pytest gets reruns through the separate pytest-rerunfailures plugin, which reports rerun outcomes and offers a fail-on-flaky option.
A passing retry is useful evidence, not a fix. Preserve attempts and artifacts locally or in CI; add shared aggregation when you need cross-run trends and ownership. For runner-specific guidance, see Playwright, Jest, and pytest.
Key features:
- Immediate evidence — observe failures and successful retries without a separate platform.
- Runner-specific policies — reporting and exit behavior vary; configure them deliberately.
- No additional software license fee — extra executions still consume time and CI resources.
Best fit: teams starting with local or CI evidence before deciding whether shared history and workflow automation justify another service.
Trade-offs: retries can mask failures if attempts are discarded. A runner report alone is not a durable shared history service, and a converted report may lose retry details. Check what your chosen reporter preserves.
Comparison: flaky-test debugging tools at a glance
| Tool | Signal and main evaluation boundary |
|---|---|
| Qualflare | Retry visibility and historical minority scoring; conditional AI analysis. Upload/analysis allowances are separate; no CI quarantine. |
| BuildPulse | JUnit-based history, impact ranking and quarantine. Verify CI setup; Test Intelligence trial differs from free runner minutes. |
| Trunk | Configurable monitors and CI quarantine. Tests keep running; workflow setup must preserve other failures. |
| Datadog | Flaky/new-flaky signals and captured CI context. Verify runner instrumentation and feature support. |
| CircleCI | Same-commit mixed results within 14 days. Needs JUnit uploads and an eligible organization. |
| Cypress Cloud | Retry-based flake signals over recorded runs. Cypress-specific; flake management requires Team or higher. |
| Develocity | Retry and cross-build signals with build diagnostics. Feature coverage varies by agent; selection is a separate capability. |
| Framework retries | Immediate attempt evidence with runner-specific reporting. Retain results separately for shared history. |
How do you debug a flaky test?
Use the tool to preserve evidence and narrow the investigation:
- Confirm the inconsistent outcome. Compare attempts and history under comparable code, inputs and environments. Mixed results on one commit can reveal an intermittent application defect. See how to detect flaky tests automatically.
- Investigate before changing the gate. Inspect failure details and assess the affected behavior. If temporary quarantine is appropriate, keep the test running, assign an owner and set a review date.
- Isolate the cause. Check timing, shared state, test ordering and environment differences, including why a test can pass locally but fail in CI. Inspect application code as well as the test.
- Fix the cause. Depending on the evidence, correct a race, wait for an observable condition, isolate test data or remove an unintended dependency. Treat clusters as investigation leads.
- Verify across relevant conditions. Repeat the affected scenario and observe later runs before restoring blocking status. More passing evidence increases confidence; it does not prove a rare failure is impossible.
For the full method behind these steps — causes, quarantine policy, and prevention — see the complete guide to flaky tests. This list is about the tools that automate the detection and tracking; the guide is about the practice.
Which tool should you choose?
There’s no single best tool here — the right pick depends on where your tests run and what’s costing you the most. Match the tool to your situation:
| If your situation is… | Start by evaluating |
|---|---|
| Supported reports across several frameworks; historical scoring and failure investigation | Qualflare, including analysis limits |
| Detection, impact ranking and quarantine from CI reports | BuildPulse |
| Configurable detection and quarantine in CI/PR workflows | Trunk Flaky Tests |
| You’re standardized on Datadog and want detection in the same observability stack | Datadog Test Optimization |
| An eligible CircleCI organization with JUnit results | CircleCI Test Insights |
| Recorded Cypress runs with retries | Cypress Cloud, checking plan limits |
| Build diagnostics and test analytics together | Develocity, checking agent-specific features |
| You need attempt-level evidence today | Framework retry reporting, locally or in CI |
Evaluate a representative report you already have. Can you see the initial failure and successful retry, retain the relevant artifacts, and match the same test in later runs? Check flaky-test history by environment and branch where appropriate. Then ask who owns the fix, whether the product changes CI gates, and how you will review a quarantined test. A useful first local report can precede CI integration; neither should discard the attempts that explain a green final result.
To evaluate reporting and historical scoring on your own suite, start free with Qualflare and upload a supported local or CI report. Inspect its results and any retained retry evidence, then compare subsequent launches. Historical case classification needs enough results; named clusters and launch analysis have their own eligibility and allowance limits. One upload is a starting point, not a promise of an immediate flaky-case score or root-cause diagnosis.
Frequently asked questions
What is the best tool for debugging flaky tests?
Choose by runner, retained evidence and the action you need. Qualflare combines reporting, historical scoring and conditional AI analysis; BuildPulse and Trunk also offer quarantine. CircleCI, Cypress Cloud, Datadog and Develocity suit their supported integrations. Compare retry details, cross-run history, debugging context and plan limits using results from your own suite.
How do flaky test detection tools work?
Some classify a failed-then-passed retry as flaky. Others also compare outcomes across runs, using stable test identity and tool-specific rules for commits, environments, windows and thresholds. These signals identify inconsistent behavior, not its root cause. Failure rate, retry-based flake rate and historical flaky scores measure different things.
Can you detect flaky tests for free?
Yes. Playwright can label failed-then-passed retries as flaky, Jest can log errors before retrying, and the separate pytest-rerunfailures plugin reports rerun outcomes. Reporting and exit policies differ by runner. These tools have no additional software license fee, but retries consume execution time. Shared history requires retained reports or an aggregation service; free plans and time-limited trials have different limits.
Do test retries fix flaky tests?
No. Retries reveal whether a failure repeats; they do not remove its cause. Some configurations allow a successful retry to make the build pass, while others fail on flaky outcomes. Keep attempts and debugging artifacts, investigate test and application code, and use later runs to assess whether the fix holds.
What is the difference between flaky test detection and quarantine?
Detection identifies inconsistent outcomes. Quarantine changes whether selected failures block CI; in systems such as Trunk, the tests keep running and reporting. It preserves observation while removing blocking protection. Assign an owner and review date, and verify the cause before quarantining. Qualflare reports flakiness but does not quarantine or rerun tests.
How common are flaky tests?
Prevalence depends on the suite, environment and definition. Google documented its experience in 2016, but one historical suite is not a current industry benchmark. Measure retry-based flaky runs, final failures and recurring flaky cases separately, with a stated time window and denominator.
Sources
- Google Testing Blog — Flaky Tests at Google and How We Mitigate Them (2016)
- Martin Fowler — Eradicating Non-Determinism in Tests
- Qualflare — Detect and fix flaky tests
- Qualflare — Project settings and retention
- Qualflare — Launch analysis
- Qualflare — Failure clusters
- BuildPulse — Flaky test detection and quarantine
- BuildPulse — API documentation
- Trunk — Flaky test detection
- Trunk — Quarantining flaky tests
- Datadog — Flaky test management
- CircleCI — Test Insights
- CircleCI — Upload test results
- Cypress Cloud — Flaky test management
- Develocity — Flaky test detection guide
- Develocity — Flaky tests detection
- Develocity — Predictive Test Selection
- Playwright — Test retries
- Jest — jest.retryTimes
- pytest-rerunfailures — Plugin documentation


