Test Report Formats Compared: JUnit XML, CTRF, TRX, TAP

Choosing a test report format feels like a small decision until you try to answer a question the format cannot represent. Then it becomes the whole problem, because the information was discarded at write time and no amount of downstream cleverness recovers it.

This is a capability comparison of the seven formats that actually turn up in practice, organised around the two questions that matter: does it have a real specification, and what can it express?

The capability matrix

Format	Real schema?	Retries	Flaky flag	Attachments	Steps	History
JUnit XML	No — 3 competing XSDs	Surefire dialect only	Surefire dialect only	No	No	No
CTRF	Yes, JSON Schema	Yes, first-class	Yes, defined	Yes	Yes	No
Allure	Source-defined	Inferred at build	Yes	Yes	Yes, nested	Yes, via history dir
TRX	No public XSD	No	No	Yes, as file paths	No	No
TAP 13/14	Yes, prose spec	No	No	No	Subtests (14 only)	No
xUnit.net v2	No published XSD	No	No	No	No	No
Cucumber Messages	Yes, schema files	No	No	Yes	Yes	No

Two patterns fall out immediately. Almost nothing has a real schema — the industry runs on formats defined by whatever the reference implementation happens to write. And almost nothing can express a retry, which is remarkable given that retrying tests is close to universal practice in CI.

JUnit XML: universal and unspecified

The default everywhere, and there is no official specification for it. It began as output from Apache Ant’s JUnit task and spread by imitation.

What is worth adding to the usual account is that there are at least three competing XSDs in circulation, and they disagree on real details — such as whether classname is required on a testcase:

• Maven Surefire’s, published and precise, describing Surefire’s dialect
• Jenkins’, in the jenkinsci/xunit-plugin repository
• Ant / “Windy Road”, the oldest lineage

Notably, when the JUnit team needs to validate JUnit XML, they use Jenkins’ XSD — not Surefire’s. When the project that gave the format its name has to borrow someone else’s schema to check its own output, “no official specification” is not an overstatement.

One genuine surprise: Surefire’s XSD does define rerunFailure, flakyFailure and flakyError elements. So a Surefire-produced file can carry retry information. Nothing outside Surefire’s dialect is obliged to produce or understand them, which makes this a local extension rather than a capability of JUnit XML — but it is a real counter-example to the flat claim that JUnit XML cannot express a retry, and it is exactly the kind of dialect divergence that makes the format hard to consume.

The successor worth watching: the JUnit team maintains Open Test Reporting, a language-agnostic XML and HTML reporting format built with a schema and attachment support from the start. If JUnit XML is eventually replaced, there is a reasonable chance it is replaced by JUnit.

CTRF: the fidelity option

CTRF is JSON with a published Draft-07 schema, and it is the only widely-used format that treats retries as a modelled concept rather than an afterthought.

retries is an integer count. retryAttempts is an array carrying each earlier attempt’s status, duration, message, trace, output and attachments. flaky is a boolean with an actual definition — final status passed, with at least one failed attempt before it. And rawStatus preserves the producing tool’s original status before CTRF’s five-value enum normalises it, so a timeout or aborted is recoverable rather than flattened away.

The trade-off is reach. CI systems do not read CTRF natively; you need a plugin or a service.

Allure: richest model, unusual retry handling

Allure has the most expressive data model of the lot — nested steps, attachments on both results and steps, categories, severity, and a statusDetails.flaky boolean.

Its retry handling is the thing people misunderstand. Retries are not a field. Allure groups results that share a historyId within a single launch when it builds the report, and surfaces them in a Retries tab. The relationship exists in the generated report, not in any individual result file.

Allure is also the only format here with a real answer to history — and it is a filesystem convention rather than a schema feature. A generated report contains a history/ directory (history.json, history-trend.json, retry-trend.json and friends) which you copy into the results directory for the next run, retaining up to 20 reports. It works, and it means your test history lives in a folder that a CI job has to remember to carry forward.

Two caveats. There is no standalone published schema — the contract is the source code. And Allure 3 changed the history mechanism to a single appended JSONL file at a configurable path, so guidance written for Allure 2 does not transfer cleanly.

TRX: the .NET default

What dotnet test --logger trx produces. There is no public XSD — the schema file ships inside a Visual Studio installation, and the authoritative public definition is vstest’s serialization source code.

It carries outcomes, durations, start and end times, error messages and stack traces, standard output and error, and attachments via ResultFiles — which are paths on disk rather than embedded content, so a TRX file is not self-contained.

It cannot express retries, flakiness, steps, or history, and Microsoft’s own documentation confirms this is by design rather than oversight, noting that TRX and JUnit reports “keep one final result per test instead of recording every attempt.”

Worth knowing: Microsoft’s newer test platform emits TRX, HTML, JUnit, CTRF, Azure DevOps and GitHub Actions reports — so the .NET ecosystem is not betting solely on TRX.

TAP: elegant and minimal, emphasis on minimal

The Test Anything Protocol is the oldest format here and the simplest — a line-based stream of ok and not ok. Both TAP 13 and TAP 14 have real prose specifications, which already puts them ahead of most of this list.

The limits are severe, and precise:

• Subtests exist in TAP 14 only. TAP 13 has no nesting.
• YAML diagnostic blocks exist in both — and are explicitly not standardized. TAP 14 states plainly that the data structure in a YAML block “has not been standardized” and that harnesses must accept anything their YAML parser supports. There are no standard keys.
• Duration, attachments, steps and retries do not appear in either specification. Not deprecated, not optional — absent.

TAP is community-maintained with no standards body behind it, and while TAP 14 uses RFC-2119 language, nothing declares it ratified. Use it if your ecosystem is already built on it. Do not adopt it expecting to carry rich data.

The rest, briefly

xUnit.net v2 XML is genuinely distinct from JUnit XML — <assemblies> / <assembly> / <collection> rather than <testsuite> — documented in prose, with no published XSD.

NUnit3 XML is also structurally its own thing (<test-run> / <test-suite> / <test-case>), thoroughly documented, no formal XSD, and makes no reference to JUnit XML at all.

Cucumber JSON was never formally specified — the unofficial schema project openly admits “the accuracy of the schema leaves much to be desired” — and is now deprecated in favour of Cucumber Messages, an NDJSON format that does ship real schema files. Legacy Cucumber JSON is now generated from Messages. If you are on Cucumber, Messages is the format to target.

Playwright’s JSON reporter is better specified than it looks. playwright.dev documents almost nothing about its shape, but the contract is a maintained public TypeScript definition — JSONReport, JSONReportSuite, JSONReportSpec, JSONReportTest, JSONReportTestResult — shipped with the package. Not JSON Schema, but a real, versioned contract.

How to choose

The genuinely useful framing is that these formats sit on a trade-off between reach and fidelity, and no current format gives you both.

• Optimising for reach: JUnit XML. Every CI system reads it. Accept that retries and attachments are lost at write time.
• Optimising for fidelity: CTRF if you want a schema and portable retry data; Allure if you want the richest report and are willing to manage a history directory.
• In a specific ecosystem: use its native format — TRX for .NET, Cucumber Messages for Cucumber, Playwright’s JSON for Playwright — and convert outward when you need to.

The most common sensible answer is emit two: JUnit XML so the CI system’s built-in display works, plus a richer format for whatever actually analyses the results. Two artifacts cost almost nothing and the alternative is discarding information you cannot get back.

And note what no format in this table does: none of them carry history except Allure, and Allure’s is a directory you have to remember to copy. Every question that spans runs — is this test flaky, is this failure new, is this suite degrading — lives outside the file format by construction. A report describes one execution; comparison requires storage.

For transparency: Qualflare is our product, and its CLI parses 26 of these formats, which is where the sharper details above come from. It does not run your tests. Choosing a format is a decision you should make on reach and fidelity, independently of what any vendor reads — and if you are only trying to make a report easier to get to rather than easier to analyse, that is a simpler and cheaper problem.

Frequently asked questions

Which test report format should I use?

For reach, JUnit XML — every CI system reads it, and that matters more than elegance. For fidelity, CTRF, because retries and flakiness are first-class fields rather than improvised conventions. Many teams emit both: JUnit XML for the CI system’s built-in display, and something richer for whatever actually analyses the results.

Which test report formats have an official schema?

Very few. CTRF publishes a JSON Schema and Cucumber Messages ships schema files. JUnit XML has no official specification and at least three competing XSDs. TRX has no public XSD — the schema file ships inside a Visual Studio installation. xUnit.net v2 XML and NUnit3 XML are documented in prose only. Allure’s format is defined by its source code rather than a published schema.

Can JUnit XML express a retry?

Not in the base format, but one dialect does. Maven Surefire’s XSD defines rerunFailure, flakyFailure and flakyError elements, so a Surefire-produced file can carry retry information. Nothing outside Surefire’s dialect is required to produce or understand those elements, so it is a local extension rather than a property of JUnit XML.

Does Allure store test retries?

Not as a field. Allure infers retries when it builds the report by grouping results that share the same historyId within a single launch, which then appear in the report’s Retries tab. Flakiness is a separate boolean on statusDetails. So the retry relationship exists in the generated report rather than in any individual result file.

Is TAP still worth using?

Only if you are already in an ecosystem built on it. TAP is a simple, elegant line-based protocol, but neither the TAP 13 nor the TAP 14 specification mentions duration, attachments, steps, or retries, and the YAML diagnostic blocks that could carry extra data are explicitly not standardized. Anything you add is a private convention.