pytest test reporting

Generate JSON for your scripts, keep HTML for inspecting a run, and use JUnit XML for CI integrations. When you need shared results and history, upload local or CI runs to Qualflare. Start with the report you need, then choose the upload path that fits.

Generate a pytest JSON report

Install pytest-json-report in the same Python environment as your tests, then enable it with --json-report:

python -m pip install pytest-json-report
python -m pytest --json-report --json-report-file=report.json

This writes report.json with a session summary, test IDs, outcomes, timing and available captured output. Omit --json-report-file to use the default .report.json. Use --json-report-summary only when you do not need per-test details. Keep distinct filenames or archive the files if you need to retain previous runs.

To keep JSON and upload XML from the same run, generate both:

# Keep JSON for your scripts and emit XML for Qualflare in the same run
python -m pytest --json-report --json-report-file=report.json \
  --junitxml=pytest-report.xml

Then follow the JUnit XML upload path below. Local reports remain useful alongside CI; choose the format for the person or tool reading it.

Choose the report for the job

pytest-cov answers a different question: which code your tests exercised. pytest-sugar changes terminal output. Both can complement your report; adding --junitxml does not itself turn off terminal plugins. For a portable pytest-html report, use --html=report.html --self-contained-html with that plugin installed; check any externally linked assets before sharing.

Send pytest results to Qualflare

Here is a real public run from the qualflare-pytest project’s own tests. Its overview shows six passing cases, including one that passed after a retry. Shared launch history complements the files you keep locally.

1. Install the CLI and save a login

Start with a Qualflare project, a project access token, and the Identifier of an existing environment. In these examples it is staging; replace it with yours, not the environment’s display name. qf is a separate binary, not a Python package.

# Install qf separately from your Python packages
npm install -g @qualflare/cli
# Or: brew install qualflare/tap/qf

# Save a local alias; enter your project access token at the hidden prompt
qf login my-project

my-project is a saved local login alias; the token selects the project. See CLI installation and authentication for the other install and login options.

2. Choose the native reporter or JUnit XML

Native reporter. Use qualflare-pytest for per-attempt retry history, steps, attachments, labels, links, tags and priority. It requires Python 3.9+ and pytest 7.0+; use qf 0.1.24 or later for file attachments. Configure the reporter in your existing pytest config file:

# pytest.ini
[pytest]
qualflare_environment = staging
qualflare_output_dir = qualflare-results

# pyproject.toml — use this instead if it is your pytest config file
[tool.pytest.ini_options]
qualflare_environment = "staging"
qualflare_output_dir = "qualflare-results"

Use one configuration file, not both. For setup.cfg, the section is [tool:pytest]. The ini setting takes precedence over QUALFLARE_ENVIRONMENT.

python -m pip install qualflare-pytest
python -m pytest
qf my-project collect ./qualflare-results --environment staging

The plugin registers itself on installation when pytest plugin auto-loading is enabled. It writes qualflare-results without making network calls; qf uploads the results. Run the collect command even if pytest reports failing tests. An explicit --environment on collect selects the upload environment.

For repeat local runs, the CLI selects the newest native run ID in the directory. See run selection and result directories. The reporter guide also covers limitations: captured stdout/stderr are not attached automatically, and xfail/xpass map to skipped/passed, including the current strict-xfail limitation.

JUnit XML: upload without the native plugin

pytest writes JUnit XML without an extra plugin. It carries results, durations and errors, and may include properties or captured output. It does not provide the native reporter’s steps, attachment workflow or full per-attempt error history.

# Built into pytest: no reporting plugin needed
python -m pytest --junitxml=pytest-report.xml
qf my-project collect pytest-report.xml --format python --environment staging

If you already generated XML alongside JSON above, collect that XML file without rerunning tests. See the pytest-aware XML format.

Upload pytest results when CI tests fail

This GitHub Actions workflow uses the native reporter. It assumes your test dependencies are in requirements.txt; replace that install command with your project’s own setup if needed. Save your project token as the QF_TOKEN Actions secret and use your environment Identifier in both the reporter configuration and upload command.

# .github/workflows/pytest.yml
name: pytest
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
      - name: Install test dependencies
        run: |
          python -m pip install -r requirements.txt
          python -m pip install qualflare-pytest
      - name: Run pytest
        env:
          QUALFLARE_ENVIRONMENT: staging
          QUALFLARE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
        run: python -m pytest
      - name: Upload results to Qualflare
        if: ${{ !cancelled() && hashFiles('qualflare-results/*.json') != '' }}
        env:
          QF_TOKEN: ${{ secrets.QF_TOKEN }}
          QF_BRANCH: ${{ github.head_ref || github.ref_name }}
          QF_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
        run: |
          npm install -g @qualflare/cli
          qf login ci --force
          qf ci collect ./qualflare-results --environment staging

The upload condition runs after failed tests when a report exists, and skips cancelled runs. The test step keeps its nonzero exit status, so a successful upload does not turn a failed job green. If setup or collection fails before a report is written, the upload is skipped and the original failure remains. QF_TOKEN is read by qf login ci --force; ci is the saved alias used by collect.

XML adaptation. Keep checkout, Python, Node and your test dependency installation. Omit the qualflare-pytest install and replace the run/upload steps with these:

# XML alternative: replace the Run pytest and Upload steps above
- name: Run pytest with JUnit XML
  run: python -m pytest --junitxml=pytest-report.xml
- name: Upload XML to Qualflare
  if: ${{ !cancelled() && hashFiles('pytest-report.xml') != '' }}
  env:
    QF_TOKEN: ${{ secrets.QF_TOKEN }}
    QF_BRANCH: ${{ github.head_ref || github.ref_name }}
    QF_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
  run: |
    npm install -g @qualflare/cli
    qf login ci --force
    qf ci collect pytest-report.xml --format python --environment staging

Both examples record the pull request’s head branch and commit. For fork pull requests, Actions does not supply repository secrets; run authenticated uploads only where your workflow has access to the project token. See the CI guide for GitLab, Jenkins and other providers.

pytest-xdist workers and separate CI jobs

Workers inside one run. With the native reporter, pytest -n auto sends results from xdist workers to the controller. The controller writes one report for the run, preserving each test’s worker index:

# Native reporter installed and configured as above
python -m pip install pytest-xdist
python -m pytest -n auto
qf my-project collect ./qualflare-results --environment staging

Separate CI jobs. Save every job’s results as a distinct artifact, gather all reports and attachments into one results directory, then authenticate and collect once. Give native reports a shared QUALFLARE_RUN_ID for that workflow attempt, and rerun all shards together. Separate collect calls create separate launches. Follow the gather-and-upload guide for this multi-job pattern.

For separate jobs producing XML, give each file a distinct name and collect the files in explicit order:

# Gather separate CI jobs' XML files first, then upload them together
qf ci collect shard-0.xml shard-1.xml shard-2.xml \
  --format python --shard --environment staging

Authenticate the final upload job first. --shard assigns indexes by file order, starting at zero, and needs at least two files. It does not recover worker indexes from a single combined XML file.

Read parametrized cases, fixture errors and retries

Parametrized tests stay distinct. Each parameter combination has its own test ID, so test_login[chrome-admin] and test_login[firefox-admin] remain separate results:

import pytest

@pytest.mark.parametrize("browser,role", [
    ("chrome", "admin"),
    ("chrome", "guest"),
    ("firefox", "admin"),
])
def test_login(browser, role):
    ...  # your assertion here

Fixture errors are different from assertion failures. A setup or teardown exception is reported as an error. This helps distinguish a broken fixture from a failing assertion in the test body.

Retries are optional. pytest does not retry by default. With pytest-rerunfailures, the native reporter preserves each attempt’s status and errors:

# With qualflare-pytest installed: preserve per-attempt status and errors
python -m pip install pytest-rerunfailures
python -m pytest --reruns 2
qf my-project collect ./qualflare-results --environment staging

On the XML path, the pytest-aware parser can infer retry counts and a final-pass flaky result from adjacent repeated cases, but cannot reconstruct error details absent from the file. Retries within a run and pass/fail history across runs provide different evidence; keep that distinction when investigating intermittent failures.

More frameworks: Playwright, Cypress, Jest, JUnit, Android (Espresso), iOS (XCTest) and Maestro. Browse all reporting guides, compare platforms or check plans and allowances.

Frequently asked questions