CucumberJS test reporting

Gherkin exists so that a failing test reads like a sentence a product manager wrote. Then it fails in CI and you get a stack trace. Qualflare has a native cucumber-js formatter that keeps the Given/When/Then trace all the way into hosted, historical reporting: every Scenario Outline row as its own case, screenshots attached to the step that took them, AI clustering by root cause, and flaky scores from run history.

CucumberJS formatters, explained

cucumber-js ships twelve formatters, and unlike Mocha you can run several at once — --format is repeatable:

cucumber-js --format html:report.html      # the official HTML report
cucumber-js --format junit:report.xml      # JUnit XML, for CI plugins
cucumber-js --format message:out.ndjson    # Cucumber Messages — the rich one
cucumber-js --format progress-bar          # and: pretty, summary, progress, rerun,
                                           #      usage, usage-json, snippets, json

• message. Cucumber Messages as newline-delimited JSON — the richest native format any of these runners emits, carrying the full Gherkin document, every step, every attachment. This is the one to build on, and Cucumber say so themselves.
• json. The format most third-party Cucumber tools were written against — and now officially in maintenance mode, with cucumber-js’s own help text pointing you at message plus the standalone json-formatter instead. Worth knowing before you build anything new on it.
• html. The official self-contained HTML report, with steps, attachments and inline screenshots. Very good for inspecting one run; like any local file, it doesn’t survive to the next.
• junit. JUnit XML for CI plugins. The oldest common denominator, and the one that loses the most: Gherkin steps have nowhere to go in that schema.
• pretty, progress-bar, summary, progress. Terminal output, from a full colour trace down to one character per scenario.
• rerun, usage, snippets. Workflow tools rather than reports — a file of failing scenarios to re-run, where each step definition is used and how slow it is, and stubs for undefined steps.

Where the formatters stop

Be clear about this, because Cucumber is the strongest case in this comparison: the format is not the problem. Cucumber Messages already carries more structure than most tools know what to do with. The problem is that a formatter formats one run. It writes a file, and the next run overwrites it.

So the questions a BDD suite actually raises after a few months — which scenario has been flaky since the April release, which of these 30 failures share one root cause, is the checkout feature slower than it was — aren’t ones a formatter is built to answer. They need results stored over time and analyzed.

Send CucumberJS results to Qualflare

Add the formatter in cucumber.json:

// cucumber.json
{
  "default": {
    "format": ["@qualflare/cucumberjs/formatter"],
    "formatOptions": { "environment": "staging" }
  }
}

Then run and upload. The formatter makes no network calls and holds no credential — it writes a directory, and the CLI uploads it:

npm install --save-dev @qualflare/cucumberjs

npx cucumber-js                                 # writes ./qualflare-results, zero network calls
qf my-project collect ./qualflare-results       # uploads it

Because --format is repeatable, this is additive rather than a swap — keep whatever console output you already read:

# --format is repeatable, so the Qualflare formatter is additive —
# you keep the console output you already read
cucumber-js \
  --format progress-bar \
  --format @qualflare/cucumberjs/formatter

In CI that’s one added step (GitHub Actions shown — GitLab CI, Bitbucket Pipelines, and Jenkins work the same way). Authenticate the CLI once with your Qualflare access token, stored as a CI secret — see the CLI docs.

# .github/workflows/tests.yml
- name: Run features
  run: npx cucumber-js

- name: Upload results to Qualflare
  if: always() # upload even when scenarios fail — that's the point
  run: qf my-project collect ./qualflare-results

What you get on top of cucumber-js

• Gherkin steps survive the trip. Given/When/Then arrive with their keyword and status, so you see which step broke — not just which scenario.
• Scenario Outline rows as cases. Each Examples row gets its own status and history, so a flaky row is visible instead of averaged into the outline.
• AI failure clustering. When one broken step definition or a downed environment takes out 30 scenarios, Qualflare groups them by root cause — so you fix one thing.
• Flaky detection from history. Each scenario is scored from its pass/fail record across runs, which catches the intermittent ones that happened not to flake today.
• Screenshots and videos attached in place. Anything from World.attach() lands on the step that produced it, written out of band rather than inflating the report.
• History, trends & defects. Pass rate, slowest features, and flakiness over time across branches — plus a defect you can open straight from a failing scenario.

Formatters vs Qualflare

Complementary, and literally so — --format is repeatable, so keep the formatters you use locally and add Qualflare for hosted, historical CI observability.

Qualflare works the same with Playwright, Cypress, Jest, Vitest, Mocha, pytest, Go, JUnit and 20+ more frameworks. Testing a mobile app too? See our Android (Espresso), iOS (XCTest), and Maestro guides. Weighing tools? See how it compares to other test management platforms, or browse all framework reporting guides. Every reporter is open source.

Outlines, tagged retries, and nested steps

First, Scenario Outlines fan out. Each row of an Examples table runs as its own scenario and is reported as its own case, with the example values in the name. A ten-row outline is ten rows in your report — which is the point, because a flaky row is visible rather than averaged into its parent.

Second, cucumber-js has the most surgical retry flag of any runner here. Not just --retry, but --retry-tag-filter — retry only the scenarios you have already tagged as suspect, and let a genuine regression fail on the first attempt like it should:

# Retry only the scenarios you've already tagged as suspect
cucumber-js --retry 2 --retry-tag-filter "@flaky"

The formatter records per-attempt history rather than a retry count, so you can see what each attempt did. One caveat worth knowing: attempts carry their own status, duration and error, but steps, labels, tags and attachments come from the final attempt — an abandoned attempt’s step trace is discarded rather than replayed alongside it.

Third, sharding needs no merge step. Point every shard at the same output directory; each process writes a uniquely named file, and merging is driven purely by which files are in the directory:

# Every shard writes into the same directory
npx cucumber-js --shard "$SHARD_INDEX/$SHARD_TOTAL"

# once, after every shard finishes
qf my-project collect ./qualflare-results

Finally, the metadata API. Cucumber is the one framework here where steps are already the test, so qualflare.step() is for nesting — when a single step definition does several things worth timing separately:

import { Given, When } from '@cucumber/cucumber';
import { qualflare } from '@qualflare/cucumberjs';

Given('a user with valid credentials', function () {
  qualflare.label('epic', 'Authentication');
  qualflare.tag('smoke');
});

When('they log in', async function () {
  await qualflare.step('fill in credentials', async () => {
    await this.page.fill('#email', '[email protected]');
  });

  await qualflare.step('submit and verify redirect', async () => {
    await this.page.click('#submit');
    await this.page.waitForURL('**/dashboard');
  });
});

Full reference in the metadata API docs.

Frequently asked questions