CTRF: A Common Format for Test Reports

Every team that aggregates test results eventually writes the same adapter twice. JUnit XML is universal but has no specification and cannot express a retry. Each framework’s native JSON is richer and completely non-portable. So you end up converting everything into JUnit XML and losing the interesting parts, or writing a parser per framework.

CTRF — the Common Test Report Format — is an attempt to fix that with a JSON schema that keeps the parts JUnit XML drops.

It is worth knowing about. It is also worth being precise about what it is, because the project’s own framing is more expansive than the evidence supports, and a format is exactly the kind of thing where you want to know who is actually behind it before you build on it.

What a CTRF report looks like

{
  "reportFormat": "CTRF",
  "specVersion": "1.0.0",
  "results": {
    "tool": { "name": "playwright" },
    "summary": {
      "tests": 3, "passed": 2, "failed": 1,
      "skipped": 0, "pending": 0, "other": 0,
      "start": 1757404462000, "stop": 1757404501000
    },
    "tests": [
      {
        "name": "checkout applies discount code",
        "status": "passed",
        "duration": 812,
        "retries": 2,
        "flaky": true
      }
    ]
  }
}

The shape is unremarkable, which is the point. What matters is what the schema requires and what it can express.

What the schema actually requires

This is the detail most write-ups get wrong, so it is worth stating exactly. From the published JSON Schema:

Level	Required fields
Top level	results, reportFormat, specVersion
results	tool, summary, tests
results.tool	name
results.summary	tests, passed, failed, skipped, pending, other, start, stop
each tests[] entry	name, status, duration

Two things to notice. reportFormat is a constant — it must literally be the string "CTRF" — and specVersion is a semver string. A document containing only results is not valid CTRF, which is easy to get wrong if you are hand-rolling a producer, because results is the only part anyone talks about.

Everything else is optional: reportId, runId, timestamp, generatedBy, insights, baseline, and an extra escape hatch at several levels.

The status model

Five values, and only five: passed, failed, skipped, pending, other. The same enum applies to a test’s status, to each retry attempt’s status, and to each step’s status.

One honest caveat: the specification does not define what other means. It says the status must be one of the five, and defines the summary field as “the count of tests whose status is other” — and stops. Every gloss you will read about other being for outcomes that cannot be normalised is a reasonable convention, not something the spec states. If you are writing a consumer, the safe treatment is to surface other as something that needs a human look rather than folding it into either pass or skip, because a format that lets a launch containing unmapped outcomes report clean is a format that will eventually report clean on a real failure.

Alongside status sits rawStatus, and this is the genuinely thoughtful part of the design. It is optional, and it holds “the original status from source tool before normalization”. So a tool that reported timeout or aborted or errored — outcomes CTRF’s five-value enum collapses — can preserve what it actually said. A consumer that cares can recover the distinction; one that does not can ignore the field. Normalising without destroying is a hard thing to get right in an interchange format, and CTRF gets it right.

Note also that pending means pending here. In Mocha-family tooling, pending is what a skipped test fires, and consumers routinely fold it into skipped for that reason. CTRF carries skipped and pending as separate enum values, so folding them discards a distinction the format draws deliberately. Same word, different meaning, decided by the source.

Retries and flakiness as first-class fields

This is the reason to care about CTRF at all.

• retries — an integer count, minimum zero.
• retryAttempts — an array of objects, one per earlier attempt, each carrying its own attempt number, status, duration, message, trace, stdout/stderr, and attachments.
• flaky — a boolean.

The spec constrains these against each other: if retryAttempts is present, retries must be present and equal the number of entries, and the final attempt number must equal retries + 1. That is the kind of internal consistency rule that makes a format trustworthy to consume.

And flaky has a real definition rather than a vibe: a test is flaky only if its final status is passed and it had one or more failed attempts before passing. A still-failing test that was retried five times is not flaky; it is failing. That matches how test retries should be reasoned about generally, and it is refreshing to see it written into a schema.

Compare this to JUnit XML, where a retry is an improvised <property name="retries"> with no agreed name, and the whole notion of a flaky test is unrepresentable. Microsoft’s documentation makes the contrast explicitly, noting that TRX and JUnit reports “keep one final result per test instead of recording every attempt.”

Who is actually behind it

Here is where the honest read matters, because the project’s public framing and its commit history do not entirely agree.

Against calling it a standard: CTRF is written almost entirely by one person — 64 of the 67 commits on the specification repository, with three other contributors at one commit each. There is no steering group and no corporate backing. The specification repository sits at 93 stars. Every reporter is still pre-1.0, at versions like 0.0.29. The site describes the format as “created by the community”, which the commit graph does not support. The README states CTRF was released in 2023, but every dated artifact — repository creation, first commit, first npm publish — falls between 30 January and 1 February 2024.

For taking it seriously anyway: the adoption is real and substantially outside the maintainer’s control. The Playwright reporter pulls over 1.8 million npm downloads a month, with the Jest reporter above 600,000. There is an official Jenkins plugin in the jenkinsci organisation. And most tellingly, Microsoft ships a CTRF reporter in the .NET test platform — Microsoft.Testing.Extensions.CtrfReport, invoked with --report-ctrf, documented on Microsoft Learn. That is third-party validation a solo maintainer cannot manufacture. It is also, in fairness, an alpha package that Microsoft labels experimental, so it is a signal of interest rather than a commitment.

One more caveat worth knowing before you plan around it: the adopters page shows tiles for pytest, RSpec, Selenium and others, but no native pytest or RSpec reporter exists. Those languages reach CTRF by converting JUnit XML first, which means they arrive having already lost the retry detail that is the format’s whole reason to exist.

Should you use it?

A reasonable position: CTRF is worth emitting alongside JUnit XML, not instead of it.

JUnit XML remains the thing every CI system already reads, and that ubiquity is not something a better schema displaces quickly. But if your pipeline retries tests — and most do — JUnit XML throws away the retry history at the moment it is written, and no downstream tool can recover it. Emitting CTRF as a second artifact costs one reporter dependency and preserves the part that matters.

If you are on Playwright, Jest, Cypress, Mocha, WebdriverIO, Newman, Go or .NET, a native reporter already exists. If you are on pytest or RSpec, you would be converting from JUnit XML, which defeats most of the purpose — wait for a native reporter or stay put.

It is also worth watching what JUnit itself is doing here. The JUnit team has been building Open Test Reporting, a successor format with a schema from day one and native attachment support, which raises the possibility that the JUnit-XML-replacement question gets answered by JUnit rather than by CTRF. Neither has won yet.

For transparency: Qualflare is our product, and its CLI has a CTRF parser among 26 formats — which is how several of the details above are known first-hand rather than read off a spec. It does not run your tests. If you are choosing a format purely on which tools can read it, JUnit XML still wins on reach and will for a while; CTRF wins on fidelity, and that gap only matters once you care about what happened before the final attempt. Where those reports end up living is a separate problem worth solving too.

Frequently asked questions

What is CTRF?

CTRF stands for Common Test Report Format. It is a JSON format for test results, defined by a published JSON Schema, intended to give tooling a single shape to consume regardless of which test framework produced the run. Its main practical advantage over JUnit XML is that retries and flakiness are first-class fields rather than improvised property conventions.

What fields does a valid CTRF report require?

At the top level, three: results, reportFormat (which must be the string "CTRF"), and specVersion. Inside results, three more are required — tool, summary, and tests. Each entry in tests requires name, status, and duration. A document containing only results is not valid CTRF.

How does CTRF represent retries and flaky tests?

With three related fields. retries is an integer count of retries. retryAttempts is an array of objects describing each earlier attempt, carrying its own status, duration, message, trace, output and attachments. flaky is a boolean, and the spec defines it precisely — a test is flaky only if its final status is passed and it had one or more failed attempts before passing.

What does the CTRF status other mean?

The specification does not say. It lists other as one of the five permitted status values and defines the summary field as the count of tests with that status, but gives no semantics for when a producer should use it. In practice tools use it for outcomes that do not map onto the other four, but that reading is convention rather than specification.

Is CTRF a real standard?

It depends what you mean by standard. It has a published JSON Schema, an official Jenkins plugin, around 23 reporters, over 1.8 million monthly npm downloads on the Playwright reporter alone, and an implementation in Microsoft’s .NET test platform. It is also written almost entirely by one maintainer, has no steering group or corporate backing, and its reporters are all still pre-1.0. It is a project with genuine adoption rather than a multi-vendor standards effort.