Product news and testing tips.
CI uploads your HTML test report as a zip artifact that expires in 90 days, and the six-step ritual needed to look at one means almost nobody does. Playwright's report opens from a `file://` URL but its traces and screenshots fail to load; Allure's needs a real web server. The fix is to serve reports from a URL rather than ship them as files — and to accept that hosting a single run still cannot answer questions about many runs.
Key takeaways
- Reading a CI test report typically takes six steps — which is why the report goes unread.
- Playwright's HTML report does open from file://, but every attachment, trace and screenshot fails to load.
- Setting PLAYWRIGHT_HTML_DO_NOT_INLINE_ASSETS makes the file:// case worse — a completely blank page.
- Allure genuinely requires a web server; `allure open` exists precisely because file:// does not work.
- GitHub artifacts download as a zip and are retained 90 days by default.
Modern test frameworks produce genuinely excellent HTML reports. Playwright’s has a timeline, a trace viewer, per-step screenshots, and network detail. Allure’s has history, categories, and severity breakdowns. Real design work went into both.
Then CI zips them and puts them somewhere you have to go and get.
Here is the ritual for looking at one, in full:
- Notice the build is red.
- Open the workflow run page.
- Scroll to the artifacts section.
- Download a zip.
- Extract it.
- Open
index.html.
Six steps, and at step four you are already thinking “or I could just re-run the job and see if it passes”. Which is what most people do, most of the time, and it is a rational response to the friction rather than laziness. The report is excellent. The delivery mechanism means it goes unread.
Why file:// is hostile to test reports
Step six is where it gets genuinely technical, and the details are more interesting than the folklore suggests.
Browsers treat a page loaded from file:// as having a null origin. Almost every cross-file request such a page makes — fetch, XHR, module imports, sometimes even stylesheets — is blocked by the same-origin policy. This is a deliberate and correct security decision: without it, any HTML file you opened could read your filesystem.
Test reports are exactly the kind of application that assumption breaks, because they are single-page apps that load their data separately from their markup. How badly they break varies more than most people realise.
Playwright: better than its reputation, with a specific hole
The common claim is that Playwright’s HTML report cannot be opened without a server. That is not accurate, and the real behaviour is worth knowing precisely.
Opening playwright-report/index.html directly from a file:// URL works for the overview. The summary counts render, the test list renders, and clicking into a failed test shows its error message, code frame, and steps. No console errors.
This works because Playwright inlines the report data into index.html by default — the results are embedded in the document rather than fetched.
What fails is attachments. Clicking through to a trace, screenshot, video, or error-context file produces a fetch against playwright-report/data/<hash> that the browser blocks, and the request fails. So you get the pass/fail list and the error text, but not the trace — which, when you are investigating a failure, is the half you actually opened the report for.
There is a sharp edge here too. Playwright exposes a doNotInlineAssets option, and setting PLAYWRIGHT_HTML_DO_NOT_INLINE_ASSETS=1 makes the file:// case dramatically worse: the report’s CSS and JS are then loaded as separate files, the browser blocks them from the null origin, and you get a completely blank page. A setting that sounds like tidiness turns a partially-working report into nothing.
The supported path is npx playwright show-report, which serves the report on localhost:9323. Usefully, it also accepts a .zip directly — so you can point it at a CI artifact without extracting first, provided the archive has index.html at its top level. That single fact removes step five from the ritual and is worth knowing.
Allure: genuinely needs a server
Allure is the case where the folklore is correct. Its documentation is explicit that a report is a directory of files, and that the reliable way to view one is allure open, which starts a local web server.
Allure also has a generation step that catches people out: your test run writes raw result files into allure-results, and those are not a report. You then run allure generate to produce the allure-report directory, or allure serve to generate into a temporary directory and serve it in one go. Uploading allure-results as your artifact and expecting someone to open it is a common and thoroughly unrewarding mistake.
The documentation does mention browser flags that relax file access — Chrome’s --allow-file-access-from-files, Firefox’s security.fileuri.strict_origin_policy — and flags both as security risks. They are, and telling a team to browse the web with them enabled is not a reasonable answer.
The artifact side
The storage half has its own constraints, all documented.
GitHub artifacts download as a zip — the REST download endpoint supports only zip as an archive format. Recent versions of actions/upload-artifact expose an archive input you can set to false for a single unzipped file. Note also that file permissions are not preserved through a zipped upload, which occasionally matters for reports that ship an executable.
Retention is 90 days by default. Public repositories can configure between 1 and 90 days; private and internal repositories up to 400. Per-upload retention-days can shorten it but cannot exceed the limit set at the repository, organization, or enterprise level.
Ninety days sounds like plenty until you want to answer “was this test failing back in June?” — at which point the artifact is gone, and with it any chance of checking. The retention window quietly defines how far back your team can reason.
What actually works
Ranked roughly by effort:
| Approach | Effort | What it gives you |
|---|---|---|
npx playwright show-report path/to.zip | Trivial | Full local report from an artifact, no extraction |
| Publish to GitHub Pages / S3 / R2, link in a PR comment | Low | A URL that survives the run; one click to open |
| Self-hosted report server | Medium | Durable URLs, access control |
| Hosted test-reporting service | Low | The above as a product |
The property that matters in every row is the same: the report has a URL. That single change collapses six steps into one, and a report that costs one click is a report people actually read.
For the second row, the pattern is to push the generated report directory to a gh-pages branch or an object-storage bucket under a path keyed by run or commit, then have the workflow comment the link on the pull request. It is perhaps thirty lines of workflow YAML and it removes most of the pain.
There is also a small category of services built specifically around this problem — testdino and gaffer.sh both offer hosted test reports with shareable links — which exists precisely because the friction is real and widely felt.
What hosting does not fix
Here is the part worth being honest about, because it is where a lot of tooling advice stops one step too early.
Hosting fixes access. It does not fix analysis. A perfectly served, instantly loading, beautifully rendered Playwright report still describes one run — it cannot tell you a test’s flake rate, because a rate needs a denominator and one run does not have one. Put a URL on it and you have made a single run convenient to look at, which is a genuine improvement and not the same as understanding your suite.
The questions that actually come up in a stand-up are comparative:
- Has this test failed before, or is this new?
- Is it failing on main as well, or only this branch?
- Are these 30 failures one root cause or 30?
- Is this suite slower than it was a month ago?
None of these can be answered from one report, however well hosted, for the same structural reason JUnit XML cannot express them and GitHub Actions cannot display them: a report is a description of a single execution. Comparison requires storage.
And that is also why the 90-day retention default matters more than it first appears. The moment your answer to a question requires a run from four months ago, the artifact is gone — not because anyone decided that history was unimportant, but because nobody changed a default.
Full disclosure on where we sit: Qualflare is our product, and it is a test observability layer that ingests results from CI and keeps them, so cross-run questions have somewhere to be answered from. It does not run your tests and it is not a replacement for Playwright’s report or Allure’s — those are better at showing you one run than anything we would build. If your problem is genuinely “the report is hard to get to”, publishing it to static hosting and commenting the link is cheap, effective, and does not require buying anything. Do that first. The case for anything more only starts when your questions stop being about this run and start being about whether a test has always been like this.
Frequently asked questions
Can you open a Playwright HTML report without a server?
Partly. Opening index.html from a file:// URL renders the summary, the test list, and per-test error messages and steps, because Playwright inlines the report data into index.html by default. What fails is every attachment — traces, screenshots, videos and error context are fetched separately and the browser blocks those requests from a file:// origin. You get the overview but not the evidence, which is usually the half you opened it for.
Why does my Allure report show up blank or broken?
Because Allure reports must be served over HTTP. The report is a directory of files that load each other via requests the browser blocks from a file:// origin. Use allure open or allure serve, both of which start a local web server. Browser flags that relax file access exist but are documented security risks and are not a reasonable default.
How long are test report artifacts kept in GitHub Actions?
Ninety days by default. Public repositories can configure retention between 1 and 90 days; private and internal repositories can go up to 400 days. A per-upload retention-days setting can shorten this but cannot exceed the limit configured at the repository, organization or enterprise level.
What is the best way to host CI test reports?
For a small team, publishing the report directory to static hosting — GitHub Pages, S3, R2 — and posting the URL as a pull request comment covers most of the need. The important property is that the report has a URL that survives the run, so looking at it costs one click rather than a download and an extraction. Hosted test-reporting services do this as a product.
Does hosting the report solve flaky test detection?
No. Hosting fixes access, not analysis. A hosted report still describes a single run, and flakiness is a property of a test across many runs. To know whether a test is unreliable you need its outcomes stored over time and compared, which is a different capability from serving an HTML file well.
