Flaky Test Quarantine Strategy (and SLAs)

A flaky test quarantine SLA is a written deadline on a quarantined test — sized by severity, centrality, and owner capacity, not picked arbitrarily — that forces the test toward a fix, a deletion, or a logged escalation instead of letting it sit. Our complete guide to flaky tests covers quarantine as one stage in the detect-quarantine-fix lifecycle; this is the operational deep dive on that stage — sizing the window, tiering it, handling breaches, capping the bucket, and the exact quarantine syntax per framework.

What does a quarantine SLA actually control?

Quarantine moves a known-flaky test out of the build’s blocking path: it keeps running and reporting, but a failure no longer fails the pipeline. That’s necessary — flaky failures are expensive and widespread and erode trust in the suite — but a quarantined test is also invisible coverage. If the bug it would have caught is real, it ships silently.

The SLA keeps that trade-off temporary: a fixed window after which the test gets fixed, deleted, or explicitly re-quarantined, never just left alone. It’s effectively its own quality gate — a release-readiness rule about what’s allowed to sit unresolved. Entry should come from history-based flaky test detection — see how to detect flaky tests automatically — never a single red run.

How do you size the SLA window instead of guessing?

None of the teams that write publicly about this publish a day-count as the correct window — they publish the shape of the system instead. Slack tickets the owning team, posts a weekly channel summary of everything suppressed, and describes building toward auto-reinstating a test once reruns show it’s clean. Atlassian tickets with a “pre-decided due date” and reinstates after a “configured” healthy period — both left team-configurable. Meta is most explicit about consequences: tests “not fixed on time are marked as flaky,” ineligible for change-based test selection, so CI stops running them against the changes they should protect. The number is configurable everywhere; the discipline — owner, ticket, recheck, consequence — is universal.

Size your own window from three inputs, not a copied default: severity (what breaks in production if the bug behind this flake is real?), centrality (is this the only test covering that path, or is the risk covered elsewhere?), and team capacity (how many quarantined tests can the owner realistically investigate this sprint?). A window that ignores capacity just produces breaches nobody acts on.

Why should the SLA be tiered instead of one number for everything?

A single SLA for every quarantined test treats a payment-flow regression the same as a typo in a tooltip. A tiered SLA — by severity and centrality — gives each test a deadline proportional to what it protects:

Tier	Example	Suggested SLA	Escalates to
Critical	Checkout, auth, payment-webhook tests with no redundant coverage	3 business days	Owning team’s manager, same-day ping
High	Widely used feature path, flakes on >15% of runs	1 week	Team lead, pulled into current sprint
Medium	Secondary feature, moderate flake rate	2 weeks	Flagged in the next quarantine review
Low	Cosmetic or low-traffic assertion	30 days	Reviewed at cap-triage; candidate for deletion

Treat this as a starting framework, not a rule to copy verbatim — the two-week default the complete guide describes lines up with the medium tier here; critical and high tighten it, low relaxes it.

Quarantine vs. retry: why not just retry everything?

They solve different problems. Quarantine isolates a test from the release gate while still running and tracking it every build, so the flakiness stays visible and dated. Retry re-runs a failed test and counts it a pass if any attempt succeeds — it keeps the pipeline green, but a test that only passes on the second try is still flaky, and a careless retry can bury that signal completely instead of surfacing it. Retries are fine for staying unblocked in the moment; they’re not a substitute for a tracked, time-boxed quarantine, and confusing the two is how a suite quietly accumulates tests nobody realizes are unreliable. For a deeper decision framework on when retries are safe vs. dangerous, plus current config across frameworks, see test retry strategies.

What happens when a quarantine SLA is breached?

A breach shouldn’t be silent — it triggers a specific sequence:

1. The owner and their lead or manager both get notified. A ping to one person is easy to snooze; one on their manager’s desk isn’t.
2. The test surfaces at the next quarantine review — weekly, the same cadence Slack uses for its suppressed-test channel summaries — so a breach doesn’t sit for a month unnoticed.
3. Someone makes an explicit, logged call: fix it now, delete it (the same judgment call the complete guide walks through for low-value tests), or re-quarantine with a new deadline and a written reason.
4. Repeated breaches escalate harder. Meta’s version is structural: a test not fixed on time becomes ineligible for change-based test selection, so CI stops running it against changes it should protect. Yours can be softer, but “nothing happens” can’t be the answer twice.

How do you stop the quarantine bucket from becoming a graveyard?

The hub’s advice to cap quarantine at roughly 1% of the suite is the right instinct, but a flat count treats every quarantined test as equally dangerous. It isn’t — five critical-tier flakes are a bigger blind spot than forty cosmetic ones. Weight the cap by tier (a critical test counts for 3x toward the limit) so the alarm trips on risk, not just volume.

Track the bucket itself, not just its size: entry date, owner, tier, and SLA deadline, for every quarantined test. When the weighted cap is hit, stop admitting new quarantines until the backlog shrinks — that forced pause keeps quarantine a queue, not a one-way door. This is where a spreadsheet or issue-tracker filter quietly fails: nobody updates the “days remaining” column. The same historical, per-test tracking that a platform like Qualflare already applies to flakiness scoring is the discipline a quarantine bucket needs too — age and owner tracked automatically, not a column nobody touches.

How do you quarantine a test in Playwright, Jest, pytest, and JUnit?

The concept is universal; the mechanics aren’t. Here’s the current, verified syntax for each.

Playwright

// Skip entirely — doesn't run, reported as skipped
test.skip('checkout redirects to the confirmation page', async ({ page }) => {
  // ...
});

// Conditional skip, called inside the test body, with a reason
test('checkout redirects to the confirmation page', async ({ page }) => {
  test.skip(true, 'Quarantined: JIRA-4821, flaky on WebKit, SLA 2026-07-16');
  // ...
});

// fixme() — use when the test crashes the runner instead of just failing
test.fixme('legacy settings panel loads', async ({ page }) => {
  // ...
});

// Tag-based — stays in the suite, filtered at run time
test('checkout redirects to the confirmation page', { tag: '@quarantined' }, async ({ page }) => {
  // ...
});

# Blocking suite: everything except quarantined tests
npx playwright test --grep-invert @quarantined

# Quarantine bucket: only the tagged tests, run separately for visibility
npx playwright test --grep @quarantined

Jest

// Skip a single test — aliases: it.skip, xit, xtest
test.skip('renders the pricing table without layout shift', () => {
  // ...
});

// Skip a whole suite
describe.skip('legacy checkout flow', () => {
  // ...
});

Jest has no built-in reason parameter or tag filter, so teams encode the tracking ticket in the test name (test.skip('[JIRA-4821] renders the pricing table...')) and keep the SLA and owner in a separate tracker.

pytest

# Skip entirely — the test does not run
@pytest.mark.skip(reason="Quarantined: JIRA-4821, flaky ~20% on CI, SLA 2026-07-16")
def test_checkout_redirects_to_confirmation():
    ...

# xfail — the test still runs every time, keeping evidence flowing
@pytest.mark.xfail(reason="Quarantined: JIRA-4821, race condition in webhook handler", strict=True)
def test_checkout_redirects_to_confirmation():
    ...

skip and xfail aren’t interchangeable: skip doesn’t execute the test, which suits one that occasionally crashes the runner. xfail still runs it and reports XFAIL, better for tracking whether the bug is still happening. Set strict=True and an unexpected pass (XPASS) fails the build — a tripwire that forces someone to notice and formally un-quarantine the test instead of it staying marked xfail forever after it’s fixed.

JUnit 5

@Disabled("Quarantined: JIRA-4821, flaky ~15% on CI, SLA 2026-07-16")
@Test
void checkoutRedirectsToConfirmationPage() {
    // ...
}

@Disabled takes an optional reason string and works at method or class level. For a separate, non-blocking job, pair it with @Tag("quarantine") and filter in the build tool — Gradle’s useJUnitPlatform supports excludeTags 'quarantine' for the blocking run and includeTags 'quarantine' for the quarantine-only one.

How do you make a quarantined-test job non-blocking in CI?

Run quarantined tests in their own job so a red result there can’t fail the build, using GitHub Actions’ continue-on-error:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright test --grep-invert @quarantined   # blocking suite

  quarantine:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - name: Run quarantined tests
        continue-on-error: true
        run: npx playwright test --grep @quarantined           # non-blocking

Placement matters. On a step, a failure still reports the job — and workflow — as successful, which is what a visibility-only quarantine job needs. At the job level, a failure won’t block dependent jobs, but the job and overall run still show failed, so job-level continue-on-error alone won’t stop a quarantine job from tripping a red build. Either way, don’t mark quarantine a required status check in branch protection — that setting, not continue-on-error, is what actually gates a merge.

Start free with Qualflare to track quarantine age, owner, and SLA breaches from the same run history it already uses to score flakiness — instead of a spreadsheet column someone forgot to update.

Frequently asked questions

What is test quarantine?

Test quarantine moves a known-flaky test out of a build’s blocking path — it still runs and reports, but a failure no longer fails the pipeline. It’s containment, not a fix, which is why it needs an SLA.

How long should a flaky test stay in quarantine?

There’s no single correct number. Size it from severity, how central the test is, and the owning team’s realistic capacity — a payment-flow test earns a multi-day window, a cosmetic assertion can wait a month. Having a number and enforcing it matters more than which number you pick.

What’s the difference between quarantining a test and retrying it?

Quarantine isolates a test from the release gate while still tracking it every run, so the flakiness stays visible. Retry re-runs a failed test and calls it a pass if any attempt succeeds — it keeps the build green, but can bury the flakiness signal entirely.

What happens when a quarantine SLA is breached?

The owner and their lead or manager both get notified — a solo ping is easy to ignore. The test is raised at the next quarantine review, and someone makes an explicit call: fix it, delete it, or re-quarantine with a new deadline. It can’t expire silently.

How do you quarantine a test in Playwright, Jest, pytest, and JUnit?

Playwright uses test.skip(), test.fixme(), or a @quarantined tag filtered with --grep-invert. Jest uses test.skip and describe.skip. pytest uses @pytest.mark.skip (doesn’t run) or @pytest.mark.xfail (still runs, expects the failure). JUnit 5 uses @Disabled("reason"). Full syntax is in the sections above.

How do you stop a quarantine bucket from growing out of control?

Cap it, weighted by severity so critical-tier flakes trip the alarm faster than cosmetic ones. Track entry date, owner, tier, and SLA deadline for every quarantined test, and review the bucket on a fixed cadence — weekly is common. Hitting the cap should stop new quarantines until the backlog shrinks.