Newman & Postman test reporting

newman run prints a table and exits. The collection you built in Postman becomes a console scrollback that ends with the CI job. Qualflare turns those runs into hosted, historical reporting: every request as its own case with a stable identity across renames, AI clustering of failures by root cause, and flaky scores built from run history.

Newman’s built-in reporters

Newman ships five reporters, selected with -r (or --reporters). They are not exclusive — pass a comma-separated list and you get all of them from one run:

newman run collection.json -r cli        # human-readable console table
newman run collection.json -r json      # the full run summary as JSON
newman run collection.json -r junit     # JUnit XML, for CI plugins
newman run collection.json -r progress  # a progress bar, no artifact
newman run collection.json -r emojitrain # exactly what it sounds like

• cli. The default. A readable per-request table and a summary of assertions, requests and scripts. Excellent while you watch it; it produces no artifact.
• json. The complete run summary — run.executions with every request, response and assertion, plus run.stats and run.failures. Written to a path with --reporter-json-export. This is the richest thing Newman produces.
• junit. JUnit XML via --reporter-junit-export, for CI plugins that only speak that format. Lossy by comparison — the schema has no place for response times, request detail or per-assertion messages.
• progress and emojitrain. Console decoration. No artifact.

Why the JSON file isn’t reporting yet

It is one run. newman-results.json describes what happened this time, and the next run overwrites it — so “is the POST /checkout request getting slower, or failing more often than last month?” has no answer, because last month’s file is gone. API collections make this especially costly, since their failures are so often intermittent: a timeout, a rate limit, a race against a fixture. Those are exactly the failures a single run cannot classify.

For history you need something that stores results over time and analyzes them.

Send Newman results to Qualflare

Add the JSON reporter to the run you already have. Nothing in the collection or your pm.test scripts changes:

# Write the JSON reporter output Qualflare reads
newman run collection.json \
  -r cli,json \
  --reporter-json-export newman-results.json

Then upload it:

qf my-project collect newman-results.json --format newman

In CI that’s one changed line and one added step. Authenticate the CLI once with your Qualflare access token, stored as a CI secret — see the CLI docs.

# .github/workflows/api-tests.yml
- name: Run API tests
  run: |
    newman run collection.json -e staging.json \
      -r cli,json --reporter-json-export newman-results.json

- name: Upload results to Qualflare
  if: always() # upload even when tests fail — that's the point
  run: qf my-project collect newman-results.json --format newman

The if: always() line matters more here than almost anywhere. Newman exits non-zero on assertion failures, so without it the upload is skipped on exactly the runs you most wanted to record. Already emitting JUnit XML with -r junit? That uploads too, with --format junit — you just lose response times and per-assertion detail.

What you get on top of newman run

• Requests as cases with stable identity. Case identity comes from the Postman item ID, so renaming a request keeps its history — unusual, and genuinely useful.
• Response time on every case. Carried through from the execution, so slow-request trends are visible over time rather than only in the console table you already closed.
• AI failure clustering. When an expired token or a downed dependency takes out 30 requests, they group into one root cause instead of 30 separate errors.
• Flaky scoring from history. Newman has no retry mechanism to read, so history is not merely the better method here — it is the only one.
• Unverified requests surfaced. A request whose assertions were all skipped is recorded as skipped rather than quietly passing.
• History, trends & defects. Pass rate, slowest requests, and flakiness over time across branches and environments.

Newman reporters vs Qualflare

Complementary: keep the cli reporter for local runs and the raw JSON when you need to inspect a single response body, and add Qualflare for hosted, historical CI observability.

Qualflare works the same with k6, Playwright, pytest, Go, TestNG and 20+ more frameworks. Prefer reading first? See Newman & Postman test reporting in CI. Weighing tools? See how it compares to other test management platforms, or browse all framework reporting guides. Every reporter is open source.

Requests, assertions, and the green run that verified nothing

The first thing to understand is the granularity. Newman’s JSON reports run.executions — one entry per request per iteration — and each entry carries its own assertions array. One request becomes one case, with its assertions rolled up:

// One request, three assertions — one case in the report.
pm.test("status is 200", () => pm.response.to.have.status(200));
pm.test("responds under 500ms", () => pm.expect(pm.response.responseTime).to.be.below(500));
pm.test("returns a user id", () => pm.expect(pm.response.json()).to.have.property("id"));

All three pass and the case passes; any one fails and the case fails, carrying that assertion’s message as the error. This keeps the report shaped like your collection — the failing thing is a request you can open, not an anonymous assertion index.

Which leads to the uncomfortable part. A request with no pm.test at all passes — it made an HTTP call and nothing asserted anything about the result. That is Newman’s own semantics, not a quirk of the upload, and it means a fully green run can contain requests that verified nothing. Collections accumulate these quietly: someone adds an endpoint to reproduce a bug, never writes an assertion, and it sits in the suite for a year reporting success.

The parser draws the one line it can honestly draw: if a request had assertions and every one of them was skipped, it is recorded as skipped rather than passed, because the request was genuinely never verified. A request with zero assertions still passes. So treat a green Newman suite as a claim worth auditing — counting requests against run.stats.assertions is a five-minute exercise that usually finds something.

Second, identity survives renames. Most frameworks key a test on its name, so renaming it looks like one test retiring and another being born with no history. Postman assigns every item a stable ID inside the collection, and that ID is what the case is keyed on — so you can rename "login" to "POST /auth/login — happy path" and its flake history follows it. Delete and recreate the request and you do start over, correctly, because that is a new item.

Third, iterations multiply the run. Data-driven runs execute the whole collection once per row:

# Same requests, 50 rows of data — 50 iterations
newman run collection.json \
  --iteration-data users.csv \
  --iteration-count 50 \
  -r json --reporter-json-export newman-results.json

Fifty rows means each request appears fifty times, and the useful signal is the pass rate: a request failing on 2 of 50 data rows is a data problem, one failing on all 50 is a broken endpoint, and one failing on a different 2 each run is flaky. Finally, note that --bail stops the run at the first failure — good for fast feedback, but everything after the failure is absent from the file rather than passing, so a bailed run is not a complete result.

Frequently asked questions