pytest test reporting

pytest gives you a terminal summary and a --junitxml file — but that’s a file, not a dashboard, and it’s gone next run. Qualflare ingests your pytest results and turns them into hosted, historical reporting: AI clusters failures by root cause, scores flaky tests from run history, and rates each launch’s risk — across every CI run and every pytest-xdist shard.

Why pytest’s built-in output isn’t enough

pytest’s reporting is intentionally minimal: rich terminal output, a machine-readable --junitxml file, and — if you add a plugin like pytest-html — a local HTML report. All of it is per-run and local. There’s no place that collects results across CI runs, aggregates the shards from a pytest-xdist job, or tells you whether a test has been getting flakier over the last two weeks. For that you need something that stores results over time and analyzes them.

Send pytest results to Qualflare

Two steps. First, have pytest write JUnit XML — this is built into pytest, no plugin required:

# pytest emits JUnit XML natively — no plugin needed
pytest --junitxml=pytest-report.xml

Then upload that file with the Qualflare CLI. The --format python flag uses the pytest-aware parser; the CLI attaches your Git branch and commit automatically:

# Upload to Qualflare with the pytest-aware parser
qf my-project collect pytest-report.xml --format python

In CI, that’s one extra step after your test run (GitHub Actions shown — GitLab CI, Bitbucket Pipelines, and Jenkins work the same way). Authenticate the CLI once with your Qualflare access token, stored as a CI secret — see the CLI docs.

# .github/workflows/tests.yml
- name: Run pytest
  run: pytest --junitxml=pytest-report.xml

- name: Upload results to Qualflare
  if: always() # upload even when tests fail — that's the point
  run: qf my-project collect pytest-report.xml --format python

What you get on top of pytest

• AI failure clustering. When a single broken fixture or import takes down 30 tests, Qualflare groups them by root cause — so you fix one thing instead of reading 30 tracebacks.
• Flaky detection from history. Qualflare scores each test’s flakiness from its pass/fail record across runs — the accurate, retry-free method. (pytest doesn’t retry by default; pytest-rerunfailures reruns are captured too.)
• pytest-xdist aggregation. Upload each shard’s XML and Qualflare merges them into one launch — a single picture across all your parallel workers.
• Per-launch risk. Each CI run becomes a launch with a risk rating, the failing areas, and recommended next steps — a ship / don’t-ship signal that arrives with the results.
• History, trends & defects. Pass rate, slowest tests, and flakiness over time across branches — plus a defect you can open straight from a failing run.

Raw pytest output vs Qualflare

Complementary: keep --junitxml / pytest-html for local runs, add Qualflare for hosted, historical CI observability.

Qualflare works the same with Playwright, Cypress, Jest, JUnit and 20+ more frameworks (including a dedicated Playwright reporting guide). Weighing tools? See how it compares to other test management platforms, or browse all framework reporting guides.

Parametrized tests, fixture errors, and reruns

Three pytest behaviors shape what your reports look like. First, parametrization fans out: every @pytest.mark.parametrize case is reported as its own test with the parameter values baked into the test ID — so one function can legitimately be 30 rows in your report, and a flaky parameter combination (only [firefox-admin] fails) is visible instead of being averaged away:

# Each parametrize case is its own test in the report:
# test_login[chrome-admin], test_login[chrome-guest], test_login[firefox-admin] ...
@pytest.mark.parametrize("browser,role", [("chrome","admin"), ("chrome","guest"), ("firefox","admin")])
def test_login(browser, role):
    ...

Second, fixture failures are errors, not failures. When a fixture raises during setup, pytest reports the test as errored rather than failed — a different bucket in the JUnit XML. That distinction matters for triage: a hundred “errors” usually means one broken fixture (database down, bad env), not a hundred broken tests, and Qualflare’s failure clustering groups them accordingly instead of flooding you with individual rows.

Third, pytest has no built-in retries — flaky-test reruns come from the pytest-rerunfailures plugin, and the rerun attempts are recorded in the XML, which is exactly the signal flakiness scoring feeds on:

# Retries via pytest-rerunfailures — reruns land in the JUnit XML
pip install pytest-rerunfailures
pytest --reruns 2 --junitxml=results.xml

Frequently asked questions