TestCafe test reporting

TestCafe does something almost no other framework does: with quarantine mode on, it tells you which tests only passed after a retry. Qualflare treats that as what it is — a flakiness signal — and turns every run into hosted, historical reporting: unstable tests scored as flaky, screenshots and video kept with the failure, and AI clustering of failures by root cause.

Send TestCafe results to Qualflare

TestCafe ships a JSON reporter, so this is one flag on the command you already run. The --reporter flag takes a comma-separated list, so you keep your console output and write the file in the same run:

# TestCafe's built-in JSON reporter
npx testcafe chrome tests/ --reporter json:testcafe-results.json

# Keep readable console output at the same time
npx testcafe chrome tests/ --reporter spec,json:testcafe-results.json

Then upload it:

qf my-project collect testcafe-results.json --format testcafe

In CI that's one changed line and one added step. Authenticate the CLI once with your Qualflare access token, stored as a CI secret — see the CLI docs.

# .github/workflows/tests.yml
- name: Run TestCafe
  run: npx testcafe chrome:headless tests/ --quarantine-mode \
       --reporter spec,json:testcafe-results.json

- name: Upload results to Qualflare
  if: always() # upload even when tests fail — that's the point
  run: qf my-project collect testcafe-results.json --format testcafe

The if: always() line is the one people leave out. TestCafe exits non-zero on failures, so without it the upload runs only on green builds and your failure history — the entire point — stays empty.

The framework that admits its tests are flaky

Most frameworks give you no in-run flakiness signal whatsoever. Go has no retry mechanism, so there is nothing to read. TestNG has IRetryAnalyzer, but the common implementation overwrites the failed result, so a test that failed twice and passed on the third attempt reports a clean pass — the flakiness is actively erased. TestCafe is the exception:

# Quarantine mode: re-run a failing test, and record whether it settled
npx testcafe chrome tests/ --quarantine-mode \
  --reporter spec,json:testcafe-results.json

# Defaults are successThreshold=3, attemptLimit=5 — tune if needed
npx testcafe chrome tests/ \
  --quarantine-mode successThreshold=2,attemptLimit=4

TestCafe's own description is that quarantine mode exists to “eliminate false negatives and detect unstable tests”: a failing test is quarantined and repeated “until they yield conclusive results” — by default it must succeed 3 times, within at most 5 attempts. A test that needed those retries is not simply called green; it is marked unstable in the report:

{
  "userAgents": ["Chrome 129.0 / macOS 15.0"],
  "total": 214, "passed": 209, "failed": 3, "skipped": 2,
  "fixtures": [
    {
      "name": "Checkout",
      "tests": [
        { "name": "rejects an expired card",
          "unstable": true,          // ← passed, but only after a retry
          "screenshotPath": "...",
          "errs": [] }
      ]
    }
  ]
}

That single boolean is worth more than it looks. It is a framework telling you, unprompted, that a test is flaky — the exact fact most teams spend months reconstructing from CI history.

Qualflare records it structurally, not decoratively: the case gets an unstable tag and its flaky flag is set, so it counts in flaky scoring and filtering rather than just reading as flaky to whoever happens to scan the tags. Storing it as a tag alone was a real defect — the information was present and unusable.

History still adds something on top. TestCafe tells you a test was unstable in this run; it cannot tell you the test has been unstable in 11 of the last 50 runs, that it started three weeks ago, or that it only happens on one browser. In-run detection and historical scoring answer different questions, and TestCafe is the rare case where you get both.

What you get on top of the JSON report

• Unstable tests scored as flaky. The quarantine signal feeds flaky scoring directly instead of sitting in a tag.
• Screenshots and video kept with the failure. For UI tests the artifact usually explains more than the stack trace does.
• Fixture-qualified identity. “Checkout > loads” and “Profile > loads” stay two tests rather than colliding into one.
• AI failure clustering. When one broken selector or a downed environment takes out 30 UI tests, they group into one root cause.
• History across runs and browsers. The JSON records the user agents; the trend is what tells you whether a fix worked.
• Sharded jobs merged. One launch per run, not one per worker.

TestCafe's JSON reporter vs Qualflare

Note the first row: TestCafe already does the in-run part well, and Qualflare does not replace it. What it adds is the axis TestCafe has no way to see — the same test across thirty runs, branches and browsers.

Qualflare works the same with Playwright, Cypress, Selenium, Jest, Mocha and 20+ more frameworks. On the quarantine trade-off, see flaky or a real bug? and test quarantine. Weighing tools? See how it compares to other test management platforms, or browse all framework reporting guides. Every reporter is open source.

Frequently asked questions