TestNG test reporting

TestNG is one of the few frameworks that ships a real report — test-output/index.html exists without a single plugin. It is also a folder on a CI worker that is deleted when the job ends. Qualflare turns those runs into hosted, historical reporting: every @DataProvider row as its own case, AI clustering of failures by root cause, and flaky scores that survive retryAnalyzer — through a native TestNG reporter that keeps every retry attempt instead of the one the XML survives with.

What TestNG writes on its own

Run a suite and TestNG populates an output directory — test-output/ by default. This is genuinely more than most frameworks give you, and it is why TestNG users are often surprised to learn they need anything else:

test-output/
├── index.html              # the browsable report
├── emailable-report.html   # single-file summary, made to be mailed
├── testng-results.xml      # TestNG's OWN schema — not JUnit XML
├── testng-failed.xml       # just the failures, re-runnable as a suite
└── junitreports/
    └── TEST-*.xml          # JUnit-format XML — this is the one to upload

• index.html. The browsable report — suites, groups, timings, and the parameters each method was invoked with.
• emailable-report.html. A single self-contained file, designed for exactly the workflow its name suggests. Still a point-in-time snapshot.
• testng-results.xml. TestNG’s own schema, and richer than JUnit XML — it carries groups, parameters and per-invocation detail that the JUnit format has no vocabulary for. It is also non-standard, which is the trade-off.
• testng-failed.xml. A suite file containing only what failed, so you can re-run just the failures. Useful locally; a trap in CI, because re-running only failures makes a red build go green without anything being fixed.
• junitreports/TEST-*.xml. A JUnit-format copy. Less detailed, universally readable — and the file every downstream tool actually wants.

Why test-output/ isn’t enough

It is per-run and it is local. Every execution overwrites the last, so the directory only ever holds the most recent answer — “has checkoutWithExpiredCard been getting flakier this month” is not a question index.html can answer, because last month’s copy no longer exists. In CI it is worse: the directory lives on an ephemeral worker and is gone when the job finishes unless you remember to archive it, at which point you have a zip file nobody opens.

For history you need something that stores results over time and analyzes them.

Send TestNG results to Qualflare

Add the reporter as a test dependency. That is the whole setup — it registers itself through TestNG’s ServiceLoader support, so there is no @Listeners annotation, no -listener flag and no testng.xml to edit.

<!-- pom.xml -->
<dependency>
  <groupId>com.qualflare</groupId>
  <artifactId>qualflare-testng</artifactId>
  <version>0.1.0</version>
  <scope>test</scope>
</dependency>

Run your suite as usual. The reporter writes a report directory and makes no network calls; the CLI uploads it:

mvn -B test        # the reporter writes qualflare-results/
qf my-project collect ./qualflare-results

Requires TestNG 7.4.0+ and Java 11+. The floor is not arbitrary: ITestResult.wasRetried() has to exist, and the whole retry story below depends on it.

Without the reporter: uploading XML

If you cannot add a dependency, Qualflare still reads JUnit-format XML — you just get less. One detail matters, because the obvious file is the wrong one: upload the JUnit-format XML, not testng-results.xml. TestNG’s native file uses its own schema — <testng-results>, <test-method> — while the parser reads the JUnit schema of <testsuites> and <testcase>. On Maven, Surefire already writes the right thing with no TestNG configuration at all:

# Surefire's JUnit-format XML — the usual Maven path
qf my-project collect target/surefire-reports/ --format testng

# Or TestNG's own JUnit-format output, if you enabled JUnitReportReporter
qf my-project collect test-output/junitreports/ --format testng

In CI that’s one added step. Authenticate the CLI once with your Qualflare access token, stored as a CI secret — see the CLI docs. GitHub Actions is shown; GitLab CI, Bitbucket Pipelines and Jenkins work the same way.

# .github/workflows/tests.yml
- name: Run tests
  run: mvn -B test

- name: Upload results to Qualflare
  if: always() # upload even when tests fail — that's the point
  run: qf my-project collect ./qualflare-results

The if: always() line is the one people leave out. Without it the upload step is skipped whenever the test step fails, so the only runs that ever reach the platform are the green ones — and failure history, which is the entire point, stays empty.

What you get on top of test-output/

• Every retry attempt, not just the last one. A test that fails twice and passes on its third retryAnalyzer attempt arrives as one case with three attempts — two failed, one passed — and is marked flaky on the spot. The XML cannot show you that, because it does not contain it.
• Data-provider rows as first-class cases. Each invocation keeps its own status, duration and history instead of being averaged into its method.
• AI failure clustering. When one broken fixture or a downed dependency takes out 40 tests, Qualflare groups them by root cause — you fix one thing instead of reading 40 stack traces.
• Parallel and sharded runs merged. TestNG’s parallel modes and multi-job CI both land in one launch rather than one report per worker.
• History, trends & defects. Pass rate, slowest tests, and flakiness over time across branches — plus a defect you can open straight from a failing run.

TestNG’s built-in reports vs Qualflare

Complementary, and the “Partly” is deliberate: TestNG’s own XML carries group and parameter detail that the JUnit format cannot express, so keep test-output/ for local runs and add Qualflare for hosted, historical CI observability.

Qualflare works the same with JUnit, pytest, RSpec, Playwright, Go and 20+ more frameworks. Testing APIs or load? See our Newman and k6 guides. Weighing tools? See how it compares to other test management platforms, or browse all framework reporting guides. Every reporter is open source.

Retries, data providers, and the flakiness they hide

TestNG is one of the few major frameworks with a genuine in-run retry mechanism. Implement IRetryAnalyzer, attach it to a test, and TestNG re-runs the method after each failure until your analyzer says stop:

public class Retry implements IRetryAnalyzer {
    private int attempt = 0;

    @Override
    public boolean retry(ITestResult result) {
        return attempt++ < 3;   // three tries, then the failure is final
    }
}

@Test(retryAnalyzer = Retry.class)
public void checkoutWithExpiredCard() { ... }

This is useful, and it is also the single biggest reason TestNG suites under-report their own flakiness — though not for the reason usually given. The failures are not merely overwritten by the final pass. They are relabelled. TestNG delivers a retried failure to onTestSkipped with wasRetried() set, and that is what reaches the file. Here is what Surefire actually writes for one test that failed twice and passed on the third attempt:

<!-- target/surefire-reports/TEST-*.xml -->
<testcase name="flakyRecovers" ...><skipped/></testcase>   <!-- actually FAILED -->
<testcase name="flakyRecovers" ...><skipped/></testcase>   <!-- actually FAILED -->
<testcase name="flakyRecovers" .../>                       <!-- passed -->

Two skips and a pass. No <failure> element anywhere, no stack trace, no record that anything went wrong — and a genuinely intermittent test disappears into a green build. It resurfaces later, usually as a production defect nobody connected to it. Any tool reading that XML is working from a file the evidence has already been removed from.

The native reporter reads the run rather than the file. It checks wasRetried() and records the attempt as the failure it was, so the same test arrives as one case with three attempts — failed, failed, passed — flagged flaky, with both stack traces intact. You see it the first time it happens, in a single launch, instead of inferring it from a pattern across weeks of history. Retries stay useful for keeping the pipeline moving; they stop being a way to lose the signal.

Second, data providers fan out. A @DataProvider returning three rows produces three invocations, and each arrives as its own case:

@DataProvider(name = "roles")
public Object[][] roles() {
    return new Object[][] { {"admin"}, {"guest"}, {"owner"} };
}

// Three invocations, three rows in the report — each with its own
// status, duration and history.
@Test(dataProvider = "roles")
public void canOpenDashboard(String role) { ... }

That is what you want, because flakiness lives at the row level. If only the guest row is intermittent, per-row history points straight at the parameter combination that is unstable, whereas a single averaged case tells you only that “canOpenDashboard is a bit flaky”. The same applies to invocationCount, which runs a method N times independently.

Third, a note on testng-failed.xml. Re-running only the failed suite is a fine local habit and a poor CI one: the re-run reports green, the original failures are not in the uploaded result, and the build passes without anything being fixed. If you use it in a pipeline, upload both runs.

Three things JUnit XML has no words for

Beyond retries, the JUnit schema simply lacks the vocabulary for parts of how TestNG runs. These are not losses of detail; they are distinctions the format cannot make at all.

• A timeout is not a failure. TestNG raises onTestFailedWithTimeout separately, so a test that blew its time limit is a different event from one whose assertion failed. In XML both are <failure>; the reporter keeps timeout as its own status.
• A broken @BeforeClass is not a test failure. When configuration fails, the tests it guarded never run. The reporter records the configuration failure itself as [config] YourClass#setUp, so the cause is visible rather than inferred from a cluster of skips.
• Data-provider rows keep their arguments. Case identity is class#method(params), so the guest row stays distinct from the admin row across runs instead of three same-named cases that history cannot tell apart.

Optional: labels, steps and attachments

Because the reporter runs inside your tests rather than parsing their output afterwards, tests can say things about themselves that no report file could carry:

import com.qualflare.testng.Qualflare;

@Test
public void checksOut() {
    Qualflare.label("feature", "checkout");
    Qualflare.tag("smoke");
    Qualflare.priority(Qualflare.HIGH);
    Qualflare.link("https://jira/QF-1", Qualflare.ISSUE, "QF-1");

    Qualflare.step("add to cart", () -> {
        Qualflare.parameter("sku", "widget");
        Qualflare.maskedParameter("token");   // name recorded, value never
    });
}

Labels, tags, links, priority, description, parameters, nested steps and file attachments — plus maskedParameter, which records that a secret was involved without recording its value. There is no annotation and nothing to register; the API finds the running test through TestNG’s own thread-local. Nothing in it can fail your test: no method throws, and every call is inert when no reporter is listening, so the same code runs unchanged in a build that has never heard of Qualflare.

Frequently asked questions