Cucumber Test Reporting: Steps, Retries & JUnit XML

Cucumber has a reporting problem that is nobody’s fault: its data model has two levels and JUnit XML has one.

A scenario is a test — that maps cleanly. But a scenario is made of Gherkin steps, each with its own status, and there is nowhere in JUnit XML to put them. Every Cucumber reporting decision follows from that mismatch, including several that surprise people.

The formatters, and which are actually built in

cucumber-js formatters split into plugin-based and legacy class-based:

Plugin-based	Legacy class-based
html, junit*, message, pretty, progress, progress-bar, summary	json, rerun, snippets, usage, usage-json

* junit is the trap. The name is built in, but it resolves to @cucumber/junit-xml-formatter, a separate npm package you have to install. This has produced real confusion — Cucumber’s own docs once claimed a built-in JUnit reporter and users hit a missing-module error.

Output goes to a file with a colon, and formatters combine:

cucumber-js \
  --format progress-bar \
  --format "junit:reports/cucumber.xml" \
  --format "message:reports/cucumber.ndjson"

Worth knowing: rerun is registered as a formatter but is undocumented in the formatters guide.

Steps land in <system-out>

Here is the concrete answer to the two-level problem. The JUnit formatter’s README states it directly:

The JUnit XML report assumes that a test is a method on a class. Yet a scenario consists of multiple steps. To provide info about which step failed, the system-out element will contain a rendition of steps and their result.

So a scenario becomes one <testcase>, and its steps become text inside <system-out>. They are not dropped, and they are not promoted to separate test cases — they are flattened into a blob that a human can read and a machine largely cannot.

The practical consequence: any CI dashboard reading your JUnit XML can tell you “Adding items to the cart failed” but not “it failed at the Then the cart shows 3 items step” without string-parsing the captured output. If step-level reporting matters to you, JUnit XML is the wrong target and Messages is the right one.

Scenario Outlines expand properly

Good news here. Each Examples row becomes its own pickle in Cucumber’s model, and the JUnit report emits one <testcase> per row, named with a suffix:

Eating cucumbers - These are passing - #1.1
Eating cucumbers - These are passing - #1.2

That is the behaviour you want. A failing row is individually identifiable rather than the whole outline collapsing into a single red result — which also means a flaky outline row shows up as a specific test rather than as an intermittently failing group.

JSON is in maintenance mode, not deprecated

The distinction matters, and the internet routinely gets it wrong. Cucumber’s documentation says:

this formatter is in maintenance mode and won’t have new features added to it. Where you need a structured data representation of your test run, it’s best to use the message formatter. Tools that rely on this formatter will continue to work, but are encouraged to migrate to consume the message output instead.

That is maintenance mode. The JSON formatter does not appear in Cucumber’s deprecations list, and it is not going away. If you have a working pipeline built on Cucumber JSON, nothing is forcing you off it.

There is also a structural detail worth knowing: the JSON output is produced by post-processing Messages rather than by a formatter plugin observing the run. Messages is the source of truth now, and JSON is a rendering of it.

Cucumber Messages is the successor — NDJSON, one Envelope per line, with a real published JSON Schema. Message types include GherkinDocument, Pickle, TestCase, TestCaseStarted, TestStepStarted, TestStepFinished, TestCaseFinished, Attachment and Hook. It preserves the step structure JUnit XML flattens, which is exactly why it exists.

--retry, and what it does not record

cucumber-js has retry built in:

cucumber-js --retry 2 --retry-tag-filter @flaky

--retry-tag-filter is the part worth adopting — it confines retries to scenarios you have explicitly tagged, rather than blanket-retrying a suite and hiding every intermittent failure in it.

The documented behaviour is precise: “when a scenario passes on a retry, it’s treated as a pass overall in the results, although the details of each attempt are emitted so formatters can access them.”

So attempts are preserved in the stream. What does not exist is a flaky status — there is no flaky field anywhere in the Messages schema. The only signal is a boolean willBeRetried on TestCaseFinished.

That is a meaningfully weaker guarantee than it sounds. willBeRetried tells you an attempt is going to be retried; it does not tell you the scenario eventually passed after failing, which is the actual definition of a flaky test. Deriving flakiness means correlating attempts yourself — and if you are exporting to JUnit XML, that information is gone entirely, because JUnit XML cannot express a retry in the first place. Compare that to CTRF, which models retries and flakiness explicitly.

cucumber-jvm differs in two ways that matter

If you are on the JVM, two deltas:

There is no --format flag. cucumber-jvm uses --plugin (or -p). Available plugins include html, json, junit, pretty, progress, message, rerun, summary, testng, timeline, usage and teamcity.

There is no --retry. Reruns are delegated to Surefire or Failsafe via rerunFailingTestsCount, and this is a documented friction point rather than a clean equivalent — Surefire reruns by class name, which does not line up with Cucumber’s scenario-name test descriptions. There are also reported cases of the HTML formatter only including retried scenarios when a specific property is set.

So “just use --retry” advice from a cucumber-js post does not transfer.

A CI setup

- name: Cucumber
  run: |
    npx cucumber-js \
      --format progress-bar \
      --format "junit:reports/cucumber.xml" \
      --format "message:reports/cucumber.ndjson" \
      --retry 1 --retry-tag-filter @flaky

- name: Upload results
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: cucumber-results
    path: reports/

Emit both. The JUnit XML is what your CI system’s built-in display reads; the Messages NDJSON is what retains step structure and attempt detail when you need to understand a failure properly. Two files, negligible cost, and the alternative is discarding structure you cannot rebuild.

The BDD reporting tension

Worth naming, since it is the reason this is awkward rather than a bug anyone can fix.

BDD’s whole premise is that a scenario reads as behaviour — Given a state, When an action, Then an outcome — and that the steps are the meaningful units. The reporting formats the industry standardised on assume a test is an atomic method that either passed or did not.

Every Cucumber reporting decision is a negotiation between those. Steps in <system-out> is a reasonable compromise, not an oversight. Messages exists because the compromise loses too much.

Disclosure, since we have a horse in this race: Qualflare is ours, and it reads Cucumber output and analyses it across runs; setup is on our Cucumber test reporting page. It does not run your scenarios. If your current pain is just that CI shows a wall of undifferentiated scenario names, adding the message formatter and a reporter that understands it will help more than anything else, and costs nothing. A fuller comparison of what each format retains is in test report formats compared.

Frequently asked questions

How do you generate a JUnit XML report from Cucumber?

In cucumber-js, cucumber-js --format junit:reports/cucumber.xml. The junit name is built in but resolves to a separate npm package, @cucumber/junit-xml-formatter, which needs installing. In cucumber-jvm the equivalent is the --plugin flag rather than --format, since cucumber-jvm has no --format option at all.

How do Cucumber steps appear in a test report?

Not as separate test cases. The JUnit formatter’s own documentation explains that JUnit XML assumes a test is a method on a class, while a scenario consists of multiple steps — so to show which step failed, it renders the steps and their results as text inside the system-out element. Cucumber Messages preserves the step structure properly.

How does a Scenario Outline appear in reports?

As one test case per Examples row. Each row becomes its own pickle in Cucumber’s model, so the JUnit report emits a separate <testcase> for each, named after the scenario with a row suffix such as #1.1 and #1.2. That is what you want — a failing row is identifiable rather than collapsing the whole outline into one result.

Is the Cucumber JSON formatter deprecated?

No — it is in maintenance mode, which is a weaker statement and the one Cucumber actually uses. Its documentation says the formatter will not have new features added, and that tools relying on it will continue to work but are encouraged to migrate to the message output instead. It does not appear in Cucumber’s deprecations list.

Does Cucumber mark retried scenarios as flaky?

No. cucumber-js --retry treats a scenario that passes on a retry as a pass overall, and emits the details of each attempt so formatters can access them — but there is no flaky status anywhere in the Messages schema. The only signal is a boolean willBeRetried on the TestCaseFinished message, so any flakiness classification has to be derived downstream.