JUnit (Java) test reporting

Generate JUnit reports with Maven Surefire or Gradle, keep the local evidence, and choose what to collect. Existing XML carries outcomes; the optional native reporter adds richer evidence where supported. Qualflare adds hosted, historical reporting across uploaded runs. Retry details depend on the producer and import path; AI analysis has additional requirements.

Generate and keep your local JUnit reports

Start with a build that already discovers and runs your tests. Surefire normally writes XML and text results; Gradle’s configured Test tasks normally produce XML and HTML. Paths can be changed or reports disabled in the build configuration. A compile failure may happen before any test report exists.

# Run the configured tests with your project's build tool
mvn test       # Surefire: target/surefire-reports/TEST-*.xml
# or, using the project's Gradle wrapper
./gradlew test # XML: build/test-results/test/TEST-*.xml
               # HTML: build/reports/tests/test/index.html

Maven HTML is a separate reporting step: the Surefire Report Plugin reads existing XML with surefire-report:report-only without rerunning tests. Configure that plugin and its output location rather than assuming mvn test also generates a site. Maven Failsafe integration tests normally write target/failsafe-reports/TEST-*.xml during a configured mvn verify build. Gradle custom test tasks use their own task-named directories.

Local aggregation is possible: Surefire Report supports aggregate reporting, and Gradle’s Test Report Aggregation plugin combines configured JVM subprojects into HTML. Archive the XML/HTML you need before cleaning outputs or losing an ephemeral runner. A hosted service is optional when you want shared history across successive runs.

Send JUnit results to Qualflare

Create a Qualflare project and obtain its project token, then install the CLI and save credentials. my-project is a local identifier; the token selects the destination project. Run qf from your repository when you want automatic Git metadata; use explicit metadata flags where a runner has no usable Git checkout.

npm install -g @qualflare/cli
qf version
qf login my-project # paste a project token at the hidden prompt

Option 1: add the native JUnit Platform reporter

Version 0.2.0 requires Java 11+ and JUnit Platform 1.12+ / Jupiter 5.12+. Your build still needs a compatible test engine and launcher. The reporter registers through the Platform’s ServiceLoader mechanism; it does not replace test discovery or upload anything itself.

<!-- pom.xml -->
<dependency>
  <groupId>com.qualflare</groupId>
  <artifactId>qualflare-junit5</artifactId>
  <version>0.2.0</version>
  <scope>test</scope>
</dependency>

Keep the runtime versions aligned with your build. For Gradle, the dependency is testImplementation("com.qualflare:qualflare-junit5:0.2.0"); configure useJUnitPlatform() on the relevant Test task. Use a fresh native output directory so a later collection cannot pick up JSON from a previous run:

# Bash: fresh output, separate from your existing XML/HTML reports
export QUALFLARE_OUTPUT_DIR
QUALFLARE_OUTPUT_DIR=$(mktemp -d "${TMPDIR:-/tmp}/junit-native.XXXXXX")
test_status=0
mvn test || test_status=$?
printf 'Test exit code: %s\n' "$test_status"

# Inspect locally first; no upload with --dry-run
qf my-project collect "$QUALFLARE_OUTPUT_DIR" --dry-run --output json
# After checking the results, upload this directory once:
# qf my-project collect "$QUALFLARE_OUTPUT_DIR"

The example saves the test exit code for inspection; an automation script must still return that failure. For Gradle, substitute ./gradlew test --rerun-tasks: force execution so an up-to-date or cached task does not leave the fresh native output directory empty. Check names, counts and outcomes before uncommenting the upload. Launcher-level reporting includes JUnit 4 only when it runs through a compatible Vintage engine; it does not turn arbitrary JVM tests into Platform tests. See the reporter’s README.

Steps and labels are not automatic: use the optional QualflareExtension and explicit metadata API. Attachments must be published by the test, and images require qf 0.1.24+. The reporter documents same-JVM Surefire retry history, but its measured Gradle test-retry case forks new JVMs and loses per-attempt history. Metadata comes from the final attempt; a container setup failure can produce a synthetic failure rather than identify every test it prevented. Read the version-specific limitations.

Option 2: collect the XML you already have

This path needs no Qualflare test dependency. Use the XML from the run you intend to report, not a directory containing old outputs. The example is for one Maven module; for Gradle’s default test task, use build/test-results/test/TEST-*.xml. Only collect JUnit-compatible XML, not every file ending in XML.

The --format junit flag selects the JUnit-XML parser. A dry run checks local parsing without an upload. Choose native JSON or XML for the same execution; uploading both is not a deduplication strategy.

# Inspect XML from this run locally first
qf my-project collect "target/surefire-reports/TEST-*.xml" --format junit --dry-run --output json
# Then upload once, after checking counts, names and outcomes
qf my-project collect "target/surefire-reports/TEST-*.xml" --format junit

In CI, preserve the failing test step and collect separately. This GitHub Actions steps: excerpt cleans default Maven outputs before testing, retains available XML, and authenticates using a repository secret. Use your project’s Java version, Maven configuration and QUALFLARE_TOKEN secret. GitLab CI, Bitbucket Pipelines and Jenkins need equivalent failure/artifact handling with their own syntax; see the CLI docs.

# steps: excerpt for a Maven project; choose Java to match your build
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
  with:
    distribution: temurin
    java-version: '17'
- uses: actions/setup-node@v4
  with:
    node-version: '22'
- name: Install collection CLI
  run: npm install -g @qualflare/cli
- name: Clean prior Maven outputs
  id: clean
  run: mvn -B clean
- name: Run tests (failure must fail this job)
  id: tests
  run: mvn -B test
- name: Retain available XML, including failed tests
  if: always() && steps.clean.outcome == 'success' && steps.tests.outcome != 'skipped'
  uses: actions/upload-artifact@v4
  with:
    name: junit-xml
    path: '**/target/surefire-reports/TEST-*.xml'
    if-no-files-found: warn
- name: Collect available results
  if: always() && steps.clean.outcome == 'success' && steps.tests.outcome != 'skipped'
  shell: bash
  env:
    QUALFLARE_TOKEN: ${{ secrets.QUALFLARE_TOKEN }}
  run: |
    test -n "$QUALFLARE_TOKEN"
    qf login my-project "$QUALFLARE_TOKEN" --force
    # Bash, from the reactor root after a fresh Maven test run
    reports=()
    while IFS= read -r -d '' file; do
      reports+=("$file")
    done < <(find . -type f -path '*/target/surefire-reports/TEST-*.xml' -print0)
    if (( ${#reports[@]} == 0 )); then
      echo "No Surefire XML from this run; collection stopped." >&2
      exit 1
    fi
    qf my-project collect "${reports[@]}" --format junit

No continue-on-error is used: a test failure still fails the job. If cleaning fails, collection is skipped; if compilation fails before producing XML, the no-files guard stops collection. Adapt cleaning and discovery together when you customize report locations. The recursive ** above belongs to GitHub’s artifact action; it is not supported as a recursive qf glob. Keep raw artifacts according to your CI retention policy.

What you get on top of JUnit XML

• Failure grouping for investigation. Eligible projects can group similar failures. Suggested causes need verification; a cluster does not prove that one fix resolves every failure.
• Historical flaky scoring. Matching test identities build evidence over successive launches. Defaults require at least 10 results in the last 25 launches and at least a 30% minority pass/fail outcome; project Quality settings can change these thresholds.
• One collection across modules. Pass the intended files to one collect command for one launch. Separate uploads are separate launches unless you deliberately use supported run/shard grouping.
• Launch analysis when eligible. Analysis can summarize failures and suggest next steps. Risk is derived from the pass rate of tests that ran, not a ship/don’t-ship verdict or proof that missing tests were covered.
• Shared history and trends. Keep identities and branch metadata consistent so teams can compare matching cases across uploaded runs.

Launch analysis requires project AI enabled, an active subscription and available analysis allowance; report uploads and analyses are separate allowances. Failure clustering depends on plan and capacity, and a cluster is named after it contains five failures. Historical scoring is separate from a single XML retry flag; see flaky-test analytics.

Uploaded screenshots, videos and traces have 30-day retention. Retain independent copies if you need them longer; see collection and retention.

Raw JUnit reports vs Qualflare

Complementary: keep the Surefire / Gradle reports for local builds, add Qualflare for hosted, historical CI observability.

Qualflare works the same with Playwright, pytest, Cypress, Jest, Go and 20+ more frameworks. Testing a mobile app too? See our Android (Espresso), iOS (XCTest), and Maestro guides. Weighing tools? See how it compares to other test management platforms, or browse all framework reporting guides. Chasing intermittent failures? See our guide to detecting and fixing flaky tests.

JUnit XML dialects and multi-module collection

“JUnit XML” is less a standard than a family of dialects. Maven Surefire writes one file per test class under target/surefire-reports/; Gradle does the same under build/test-results/<task>/; other producers may wrap several suites in <testsuites>. Attribute conventions differ too — what counts as an error vs a failure, where skip reasons live, whether timestamps are present:

<!-- Illustrative minimal XML: one passing case -->
<testsuite name="com.acme.CheckoutTest" tests="1" failures="0" errors="0" skipped="0" time="0.42">
  <testcase classname="com.acme.CheckoutTest" name="appliesDiscount" time="0.42"/>
</testsuite>

A multi-module Maven build usually writes a report directory inside each module. A root-only glob misses those files; qf’s glob expansion is not recursive. This Bash example finds fresh Surefire XML at any module depth, preserves spaces in paths and sends the selected files to one collection:

# Bash, from the reactor root after a fresh Maven test run
reports=()
while IFS= read -r -d '' file; do
  reports+=("$file")
done < <(find . -type f -path '*/target/surefire-reports/TEST-*.xml' -print0)
if (( ${#reports[@]} == 0 )); then
  echo "No Surefire XML from this run; collection stopped." >&2
  exit 1
fi
qf my-project collect "${reports[@]}" --format junit

For Failsafe, match */target/failsafe-reports/TEST-*.xml instead, excluding its summary file. For Gradle’s default test task, match */build/test-results/test/TEST-*.xml; include other tasks only deliberately. qf directory collection expands native JSON files, not XML. Use fresh outputs and inspect the selected set before uploading; different modules can still produce colliding class/test identities.

Retries: what the producer writes is not always what an importer reads

Surefire’s rerunFailingTestsCount can record flakyFailure/flakyError for a recovered test and rerunFailure/rerunError for failed reruns. Gradle’s mergeReruns changes how repeated executions appear in XML; it does not enable retries by itself.

The inspected qf JUnit parser reads retries or retryCount case properties and marks a passed case with a positive count as flaky. It does not map Surefire’s flaky/rerun elements into attempt history. Do not assume those fields survive XML collection. Keep the original XML for tools that understand its dialect, or use the native path where supported.

JUnit’s repeated tests run separate invocations; @RepeatedTest is not a retry-until-pass policy. The native reporter keeps repeated and parameterized invocations as separate cases. Historical flaky scoring remains a different signal and needs matching results over time.

Frequently asked questions