Test Reporting in GitHub Actions and Its Limits

GitHub Actions runs your tests. It does not understand them.

This is the single most useful thing to know about test reporting on the platform, and it is easy to miss, because the surface area looks like it should. There are red X marks next to failing tests in pull requests. There are formatted summaries on the run page. There is a whole artifacts section. It looks like a test reporting system.

It is a set of generic display primitives that third-party actions use to draw test results. GitHub’s own documentation for building and testing Python is the clearest evidence: it tells you to run pytest --junitxml=junit/test-results.xml and then upload that file with actions/upload-artifact. That is where the guidance ends. Nothing in the platform opens the file.

Understanding what the three display surfaces actually do — and the documented limits of each — tells you exactly where the ceiling is, and why almost every team eventually adds something on top.

Does GitHub Actions parse test results natively?

No. There is no native JUnit XML parsing, no test history, no per-test view, and no flakiness signal.

What you get instead is three primitives:

Surface	What it is	Scope
Check annotations	Line-anchored messages on a commit or PR diff	One run
Job summaries	Markdown rendered on the workflow run page	One run
Artifacts	Files stored alongside the run, downloadable as a zip	One run, retained by policy

Note the third column. Every one of them describes a single execution. That constraint is the whole story, and we will come back to it.

Annotations: what they are and what limits them

Annotations are the red and yellow markers that appear inline on a pull request diff. There are two entirely separate ways to create them, with different limits, and conflating the two is the most common source of confusion.

Workflow commands

The simplest path is a workflow command written to stdout:

echo "::error file=src/checkout.py,line=42::Expected 402, got 500"
echo "::warning file=src/cart.py,line=17::Deprecated helper"
echo "::notice::Coverage dropped 0.4%"

GitHub documents a limit here: 10 warning annotations and 10 error annotations per step. Note two things about that figure. It is per step, not per job or per run — so splitting work across steps raises the effective total. And the documented limit names warnings and errors only; notice is not covered by that sentence, so do not assume it behaves identically.

You will find “50 annotations per job” widely repeated online. That figure comes from a community forum post, not from GitHub’s documentation. Treat it as folklore.

The Checks API

Third-party actions generally do not use workflow commands. They create a check run through the REST API, which has its own, more generous limit:

The Checks API limits the number of annotations to a maximum of 50 per API request. To create more than 50 annotations, you have to make multiple requests to the Update a check run endpoint. Each time you update the check run, annotations are appended to the list of annotations that already exist for the check run.

That last sentence matters. 50 is a per-request batching limit, not a ceiling. An action that wants to annotate 300 failures makes six requests. There is no documented cap on the total number of annotations a check run can accumulate — which is why a well-built action can surface every failure in a large suite while a naive ::error loop silently stops at ten.

Job summaries: 1 MiB, and only 20 of them

Job summaries are Markdown you append to the file at $GITHUB_STEP_SUMMARY, rendered on the run page. They are the nicest-looking surface of the three and the one people most often overrun.

{
  echo "### Test results"
  echo ""
  echo "| Suite | Passed | Failed | Duration |"
  echo "| --- | --- | --- | --- |"
  echo "| checkout | 128 | 2 | 41.2s |"
} >> "$GITHUB_STEP_SUMMARY"

Two documented limits apply:

• Each step is restricted to a maximum size of 1 MiB. Exceed it and the upload for that step fails, and GitHub creates an error annotation in its place. The failure is loud, which is good, but it happens at the end of a run rather than up front.
• A maximum of 20 job summaries from steps are displayed per job. Matrix builds and heavily-stepped workflows hit this without warning.

1 MiB sounds generous until you try to dump a full failure table with stack traces for a suite of a few thousand tests. It is a summary surface, and it punishes you for treating it as a report.

Artifacts: the zip nobody opens

Artifacts are the general-purpose escape hatch — upload any file, download it later.

- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: test-results
    path: junit/test-results.xml
    retention-days: 30

The if: always() is not optional in practice. Without it the step is skipped when tests fail, which is precisely when you wanted the results.

Retention: artifacts and logs are retained for 90 days by default. Public repositories can configure this between 1 and 90 days; private and internal repositories can go up to 400. A per-upload retention-days can shorten that but cannot exceed the limit set at the repository, organization, or enterprise level.

Artifacts download as a zip by default — the REST API’s download endpoint only supports zip as an archive format. Recent versions of upload-artifact expose an archive input that can be set to false to upload a single file unzipped. Worth knowing: file permissions are not preserved through a zipped artifact upload.

The practical experience of artifacts is the reason this surface disappoints. To look at a failure from three days ago you download a zip, extract it, and open a file — and by then you have usually decided it was not worth it. The results exist; the friction means nobody consults them.

The community actions that fill the gap

Because the platform ships no parser, a small ecosystem does the work. All four of the main options are actively maintained as of September 2026:

Action	What it does	Notes
dorny/test-reporter	Displays results from many frameworks directly in GitHub	v3.0.0, actively pushed
mikepenz/action-junit-report	Reports JUnit results as a PR check	v6.5.0, the most frequently released
EnricoMi/publish-unit-test-result-action	Publishes unit test results, incl. PR comments	v2.24.0, mature
test-summary/action	Compact summary of results in the run	v2.6, stable but quieter lately

These are good tools and worth using. They turn an unread XML file into a readable check. What they do not do — because the primitives they are built on do not allow it — is remember anything.

What none of this gives you

Every surface above is scoped to one run. That single design fact produces the whole list of things GitHub Actions structurally cannot tell you:

• Is this test flaky? Requires the same test’s outcomes across many runs.
• Is this failure new? Requires knowing what failed last time.
• Is this suite getting slower? Requires duration history.
• Are these 40 failures one bug or forty? Requires clustering across the failure set, and ideally across runs.
• Did this test fail on main too, or only on my branch? Requires results from other runs.

A red X tells you the build failed. It cannot tell you whether it failed for the same reason as yesterday, and that is usually the question that determines what you do next.

This is not a criticism of GitHub Actions. It is a CI runner with display primitives, and it is a good one. Test history is a different product category — one that needs to store results, not just render them — and expecting a per-run display surface to provide it is a category error. The JUnit XML format itself has the same boundary: it describes one execution and has no vocabulary for retries or history.

A reasonable setup

If you want the most out of the platform without adding anything:

1. Emit JUnit XML from your test framework.
2. Upload it as an artifact with if: always() and a retention-days that matches how far back you actually look.
3. Add one of the reporter actions above so failures are annotated on the PR rather than buried in logs.
4. Write a short job summary with headline counts — a table of suite, passed, failed, duration. Keep it well under 1 MiB and remember only 20 render per job.

That gets you a genuinely good single-run experience. When the questions your team asks start with “is this still” or “has this always”, you have outgrown what the primitives can answer, and no amount of better rendering will close that gap.

Full disclosure on where we fit: Qualflare is our product, and it is a test observability layer that ingests results from CI and analyses them across runs — flaky-test scoring from history, failure clustering, per-launch risk. It does not run your tests, it is not a CI system, and it does not replace any of the actions listed above; it reads the same JUnit XML they do and keeps it. If your pain is that a single run is badly displayed, a reporter action is the cheaper and better fix, and you should start there. Our GitHub Actions integration documents the setup if the history question is the one you actually have.

Frequently asked questions

Does GitHub Actions parse JUnit XML natively?

No. GitHub Actions has no built-in test-result parsing. Its own documentation for building and testing code stops at generating a JUnit XML file and uploading it with actions/upload-artifact — nothing in the platform reads that file, renders it, or tracks it across runs. Displaying test results requires a third-party action such as dorny/test-reporter, mikepenz/action-junit-report, or EnricoMi/publish-unit-test-result-action.

How many annotations can a GitHub Actions workflow create?

The Checks API accepts a maximum of 50 annotations per API request, and annotations are appended across successive update requests, so a third-party action can create more than 50 by making multiple calls. Workflow commands are stricter: GitHub documents a limit of 10 warning annotations and 10 error annotations per step. There is no documented cap on the total number of annotations a single check run can hold.

What is the size limit for a GitHub Actions job summary?

Each step is restricted to a maximum of 1 MiB of job summary content. If a step adds more than that, the upload for the step fails and GitHub creates an error annotation instead. Separately, a maximum of 20 job summaries from steps are displayed per job.

How long does GitHub keep test result artifacts?

By default, artifacts and logs are retained for 90 days. Public repositories can configure this to anywhere between 1 and 90 days; private and internal repositories can configure it between 1 and 400 days. Individual uploads can set a shorter period with retention-days, but cannot exceed the limit set at the repository, organization, or enterprise level.

Why can’t GitHub Actions tell me if a test is flaky?

Because every surface it provides — annotations, job summaries, artifacts — describes exactly one run. Flakiness is a property of a test’s behaviour across many runs: the same test on the same code producing different outcomes. Detecting it requires storing results over time and comparing them, which is not something a per-run display surface can do regardless of how well it renders a single report.