pytest test reporting

pytest gives you a terminal summary and a --junitxml file — but that’s a file, not a dashboard, and it’s gone next run. Qualflare ingests your pytest results and turns them into hosted, historical reporting: AI clusters failures by root cause, scores flaky tests from run history, and rates each launch’s risk — across every CI run and every pytest-xdist shard. A native pytest plugin captures results as the run happens — it registers itself on install, with no flag to add.

pytest’s native reporting options, explained

pytest’s built-in output is deliberately minimal — a terminal summary and, with one flag, a JUnit XML file. Everything richer comes from plugins, and they compose: install more than one and they all activate together.

• --junitxml. Built into pytest, no plugin required. Writes JUnit-style XML — less human-readable than the alternatives below, but the format nearly every CI platform and results-ingestion tool already understands.
• pytest-html. A self-contained, single-file HTML report (--html=report.html --self-contained-html) — filterable pass/fail/skip counts, tracebacks, and optional screenshots via a hook. Good for inspecting one local run; like any local file, it doesn’t survive to the next run.
• pytest-cov. Wraps coverage.py (--cov=mypackage) to report code coverage — the percentage of lines and branches your suite executed. Worth being precise about: that’s a different question from pass/fail results, and a separate report from the other tools here.
• pytest-sugar. Changes the terminal experience while tests run — a live progress bar, colored output, failures shown immediately instead of only in the final summary. No file output; it’s a local developer-experience upgrade, not something you’d collect afterward.
• pytest-json-report. Writes a structured JSON file with more detail than JUnit XML — per-test timing, captured output, and metadata — for scripts and tools that want to parse results programmatically rather than read XML.

None of these accumulate history on their own — each run’s HTML, JSON, or XML file is independent, so “was this flakier this week than last” isn’t a question any of them can answer by themselves. A pragmatic split keeps the local terminal pleasant and CI machine-readable:

# Typical split: prettify the local terminal, keep CI machine-readable
pip install pytest-sugar   # auto-activates once installed — no flags needed
pytest                      # sugar's progress bar + colored output, locally

pytest --junitxml=pytest-report.xml   # CI: skip the prettifying, emit XML

Why pytest’s built-in output isn’t enough

pytest’s reporting is intentionally minimal: rich terminal output, a machine-readable --junitxml file, and — if you add a plugin like pytest-html — a local HTML report. All of it is per-run and local. There’s no place that collects results across CI runs, aggregates the shards from a pytest-xdist job, or tells you whether a test has been getting flakier over the last two weeks. For that you need something that stores results over time and analyzes them.

Send pytest results to Qualflare

Install the native plugin. It registers itself on install — there is no -p flag to add and nothing to import:

pip install qualflare-pytest

Run pytest as usual. The plugin writes a report directory and makes no network calls; the CLI uploads it:

pytest                                 # writes ./qualflare-results
qf my-project collect ./qualflare-results

Requires Python 3.9+ and pytest 7.0+. pytest-xdist and pytest-rerunfailures are supported but optional — the plugin works, and is tested, without either. Configure it wherever you already configure pytest:

# pytest.ini (or pyproject.toml / setup.cfg)
[pytest]
qualflare_environment = staging
qualflare_output_dir = qualflare-results

Note that qualflare-cli is a standalone Go binary, not a Python package — pip install qualflare-cli will not find anything. Homebrew and npm are the two channels.

Without the plugin: JUnit XML

If you would rather not add a dependency, pytest's built-in JUnit XML still works — you lose steps, attachments and metadata, but the results land:

# pytest emits JUnit XML natively — no plugin needed
pytest --junitxml=pytest-report.xml

Upload it with the pytest-aware parser; the CLI attaches your Git branch and commit automatically:

# Upload to Qualflare with the pytest-aware parser
qf my-project collect pytest-report.xml --format python

In CI, that’s one extra step after your test run (GitHub Actions shown — GitLab CI, Bitbucket Pipelines, and Jenkins work the same way). Authenticate the CLI once with your Qualflare access token, stored as a CI secret — see the CLI docs.

# .github/workflows/tests.yml
- name: Run pytest
  run: pytest

- name: Upload results to Qualflare
  if: always() # upload even when tests fail — that's the point
  run: qf my-project collect ./qualflare-results

What you get on top of pytest

• AI failure clustering. When a single broken fixture or import takes down 30 tests, Qualflare groups them by root cause — so you fix one thing instead of reading 30 tracebacks.
• Flaky detection from history. Qualflare scores each test’s flakiness from its pass/fail record across runs — the accurate, retry-free method. (pytest doesn’t retry by default; pytest-rerunfailures reruns are captured too.)
• pytest-xdist aggregation. Upload each shard’s XML and Qualflare merges them into one launch — a single picture across all your parallel workers.
• Per-launch risk. Each CI run becomes a launch with a risk rating, the failing areas, and recommended next steps — a ship / don’t-ship signal that arrives with the results.
• History, trends & defects. Pass rate, slowest tests, and flakiness over time across branches — plus a defect you can open straight from a failing run.

Raw pytest output vs Qualflare

Complementary: keep --junitxml / pytest-html for local runs, add Qualflare for hosted, historical CI observability.

Qualflare works the same with Playwright, Cypress, Jest, JUnit and 20+ more frameworks (including a dedicated Playwright reporting guide). Testing a mobile app too? See our Android (Espresso), iOS (XCTest), and Maestro guides. Weighing tools? See how it compares to other test management platforms, or browse all framework reporting guides. Battling intermittent failures? Read our guide to detecting and fixing flaky pytest tests.

Parametrized tests, fixture errors, and reruns

Three pytest behaviors shape what your reports look like. First, parametrization fans out: every @pytest.mark.parametrize case is reported as its own test with the parameter values baked into the test ID — so one function can legitimately be 30 rows in your report, and a flaky parameter combination (only [firefox-admin] fails) is visible instead of being averaged away:

# Each parametrize case is its own test in the report:
# test_login[chrome-admin], test_login[chrome-guest], test_login[firefox-admin] ...
@pytest.mark.parametrize("browser,role", [("chrome","admin"), ("chrome","guest"), ("firefox","admin")])
def test_login(browser, role):
    ...

Second, fixture failures are errors, not failures. When a fixture raises during setup, pytest reports the test as errored rather than failed — a different bucket in the JUnit XML. That distinction matters for triage: a hundred “errors” usually means one broken fixture (database down, bad env), not a hundred broken tests, and Qualflare’s failure clustering groups them accordingly instead of flooding you with individual rows.

Third, pytest has no built-in retries — flaky-test reruns come from the pytest-rerunfailures plugin, and the rerun attempts are recorded in the XML, which is exactly the signal flakiness scoring feeds on:

# Retries via pytest-rerunfailures — reruns land in the JUnit XML
pip install pytest-rerunfailures
pytest --reruns 2 --junitxml=results.xml

Frequently asked questions