Playwright test reporting

A practical guide to reporting Playwright results: what each built-in reporter actually does, which ones to run in CI, how to merge sharded runs, and how to add hosted, historical analysis — AI failure clustering, flaky-test scoring, and per-launch risk — on top with Qualflare.

Playwright’s built-in reporters, explained

Playwright test reporting is the process of turning raw run results — passes, failures, retries, traces — into something a team can read and act on. Playwright ships several built-in reporters, and they compose: you can run a console reporter for humans and a machine-readable one for tools in the same run.

• list / line / dot. Console output at three verbosity levels. list prints a line per test (the local default); dot prints one character per test, which keeps CI logs short and is the default on CI.
• html. A self-contained interactive report in playwright-report/ — filterable, with attached traces, screenshots, and videos. The best way to debug a single run locally.
• json. The full structured result — every test, attempt, retry, error, and attachment — written to a file you point tools at. This is the richest machine-readable output Playwright produces.
• junit. JUnit-style XML. Less detailed than JSON, but the lingua franca of CI platforms: GitLab, Jenkins, and Azure DevOps all render it natively as a test report tab.
• blob. A raw intermediate report whose only job is to be merged later — the building block for sharded runs (more below).
• github. GitHub Actions annotations: failures show up inline on the PR diff. Zero value outside GitHub, cheap to add inside it.

A pragmatic config runs lightweight reporters locally and machine-readable ones in CI:

// playwright.config.ts — typical local vs CI reporter split
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: process.env.CI
    ? [['dot'], ['json', { outputFile: 'playwright-results.json' }], ['github']]
    : [['list'], ['html']],
});

Why the built-in HTML report isn’t enough

The HTML report is excellent at what it’s designed for — inspecting one run. Its limits show up the moment you have a pipeline: the report is written to a local playwright-report folder and overwritten on the next run. There’s no history across CI runs, no aggregation of results from parallel jobs, and no way to see whether a test has been getting flakier over the last two weeks. You can archive each run’s report as a CI artifact, but a folder of 200 zip files is storage, not reporting — nothing connects run #1481 to run #1482, and nobody opens them.

For anything beyond a single run, you need a place that collects results over time and analyzes them: trend lines, flakiness scores, failure grouping. That’s the gap the rest of this guide fills — first the CI plumbing, then the analysis layer.

CI patterns: GitHub Actions, GitLab, Jenkins

GitHub Actions. Add the github reporter for PR annotations, write JSON for downstream tools, and always upload results even when tests fail — a reporting step that only runs on green builds is useless on the day you need it:

# .github/workflows/e2e.yml
- name: Run Playwright tests
  run: npx playwright test

- name: Upload results to Qualflare
  if: always() # upload even when tests fail — that's the point
  run: qf my-project collect playwright-results.json

GitLab CI. GitLab renders JUnit XML natively in the pipeline’s Tests tab, so emit JUnit alongside whatever else you run:

// playwright.config.ts — JUnit XML for CI test-report integrations
export default defineConfig({
  reporter: [['junit', { outputFile: 'results.xml' }]],
});

# .gitlab-ci.yml — JUnit output doubles as GitLab's native test report
e2e:
  script:
    - npx playwright test
  artifacts:
    when: always
    reports:
      junit: results.xml
    paths:
      - playwright-report/

Jenkins. Same idea: produce results.xml with the JUnit reporter and hand it to the classic junit 'results.xml' pipeline step. Jenkins tracks pass/fail counts per build — useful, though it knows nothing about retries or flakiness semantics.

Whatever the platform, the principle is the same: a console reporter for the log, a machine-readable file for tools, uploaded with if: always() semantics.

Sharded runs: one report from parallel jobs

Once a suite takes more than ~10 minutes, you’ll split it across machines with --shard — and instantly create a reporting problem: four jobs now produce four partial reports. Playwright’s answer is the blob reporter plus merge-reports:

# Run a 4-way shard — each job writes a blob report
npx playwright test --shard=1/4 --reporter=blob
# (repeat with 2/4, 3/4, 4/4 in parallel CI jobs;
#  upload each job's blob-report/ directory as a CI artifact)

# In a final CI job: download all blob reports, merge to one JSON, upload once
PLAYWRIGHT_JSON_OUTPUT_NAME=playwright-results.json \
  npx playwright merge-reports --reporter json ./all-blob-reports

qf my-project collect playwright-results.json

The merged output is indistinguishable from a single-machine run, so any consumer of Playwright results — the HTML viewer, your CI’s JUnit tab, or Qualflare’s CLI — works on it unchanged. One upload per pipeline, not one per shard, keeps your launch history clean.

Common Playwright reporting problems (and fixes)

• The results file is missing after a crashed run. Reporters write their output when the run finishes — if the process is killed (CI timeout, OOM), the JSON file may never be written. Set a job-level timeout above Playwright’s own globalTimeout so Playwright ends the run and reports, instead of the CI runner killing it mid-flight.
• CLI reporter config silently replaces yours. Passing --reporter on the command line overrides the config file’s reporter array rather than adding to it — a classic cause of “the JSON stopped appearing in CI.” Keep the full reporter list in playwright.config.ts and avoid the flag in CI scripts, or repeat every reporter in the flag.
• Mismatched shard reports won’t merge. merge-reports requires all blob reports to come from the same Playwright version. If shards install dependencies independently, a lockfile update landing mid-pipeline can split versions across shards — pin the version and install from the lockfile (npm ci).
• The HTML report self-opens and hangs CI. The HTML reporter’s default open: 'on-failure' tries to serve the report after a failed local run; on CI that can stall a job. Set open: 'never' when process.env.CI is set.
• Artifacts balloon to gigabytes. Traces and videos for every test add up fast. Record them only when they’re useful: trace: 'on-first-retry' and video: 'retain-on-failure' keep failure debugging intact while shrinking artifact uploads by an order of magnitude.
• Pass rates look fine but the suite is rotting. Retries hide instability: a test that passes on attempt two counts as a pass. Track the flaky status (Playwright marks retried-then-passed tests as flaky in JSON output) — if you only watch the red/green summary, flakiness compounds until the suite stops being trusted. This is the metric to alert on, not the pass rate.

Send Playwright results to Qualflare

It’s a 2-minute, two-step setup. First, add Playwright’s JSON reporter so each run writes a machine-readable results file (keep your HTML reporter for local debugging):

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Keep your existing reporters and add JSON for upload:
  reporter: [
    ['html'],
    ['json', { outputFile: 'playwright-results.json' }],
  ],
});

Then upload that file with the Qualflare CLI. There’s no per-framework configuration — the CLI auto-detects the Playwright format and attaches your Git branch and commit:

# Upload the results — the CLI auto-detects the Playwright format
qf my-project collect playwright-results.json

In CI it’s the one extra step shown in the GitHub Actions snippet above — and identical in GitLab CI, Bitbucket Pipelines, and Jenkins. Authenticate the CLI once with your Qualflare access token, stored as a CI secret — see the CLI docs. For sharded suites, run qf collect on the merged JSON from merge-reports.

What you get on top of Playwright

• AI failure clustering. When a deploy breaks 40 Playwright specs, Qualflare groups them by root cause so you triage a handful of clusters instead of 40 stack traces.
• Flaky-test scoring from your retry data. The CLI reads Playwright’s native retry counts and flaky status, and Qualflare scores each test’s flakiness across runs — so you can quarantine or fix the genuinely unstable ones.
• Per-launch risk. Every CI run becomes a “launch” with a risk rating, the failing areas, and recommended next steps — a ship / don’t-ship signal that arrives with the results.
• History & trends. Pass rate, slowest specs, and flakiness over time across branches and shards — the aggregation the HTML report can’t do.
• Context & defects. Failures keep their Playwright traces and screenshots, and you can spin up a defect straight from a failing run.

Built-in HTML reporter vs Qualflare

They’re complementary: keep the HTML reporter for local debugging, add Qualflare for hosted, historical CI observability.

Qualflare works the same with pytest, Cypress, Jest, JUnit and 20+ more frameworks. Weighing tools? See how it compares to other test management platforms, or browse all framework reporting guides.

Frequently asked questions