Product news and testing tips.
Three genuinely open-source tools do cross-run flaky detection — Allure Report, ReportPortal and CTRF's GitHub reporter — which is more than most write-ups admit. What they share is shallow history: fixed windows, a 20-report cap, or borrowed artifact retention. The gap is not detection, it is durable storage, and the Jenkins install numbers show how few teams have crossed it at all.
Key takeaways
- Allure Report (Apache-2.0) flags a test flaky using its last 5 launches — in the free tool, not just TestOps.
- ReportPortal is actively maintained; its April 2026 announcement moved SSO and LDAP to paid, not the project into maintenance.
- Jenkins' JUnit plugin is on 96.6% of controllers; its flaky-test-handler plugin on 0.43% — a 225:1 gap.
- An MIT-licensed uploader is not an open-source product — Trunk and BuildPulse both ship one over a proprietary service.
- OpenTelemetry's semantic conventions define four test attributes and have no concept of flakiness at all.
Start with a fact that frames the whole category: OpenTelemetry has no concept of a flaky test.
Its semantic conventions define exactly four test attributes — suite name, suite run status, case name, case result status — all still marked experimental. Search the entire semantic-conventions repository for “flaky” and you get zero results — there is no attribute for a flaky test, and none for a flake rate either. The industry’s standard vocabulary for observability has no word for the most common failure mode in testing.
That absence explains a lot about what follows. There is no standard to implement, so every tool invented its own approach, and the open-source ones are more capable than their reputation while stopping at a consistent place.
Which OSS tools actually score flakiness
The common claim is that flaky detection is a commercial feature. That is wrong — three genuinely OSI-licensed, self-hostable tools do it.
| Tool | Licence | Algorithm | Where it stops |
|---|---|---|---|
| Allure Report 2 & 3 | Apache-2.0 | Failed in last 5 launches, passed since, failing now | You plumb history yourself; caps at 20 reports |
| ReportPortal | Apache-2.0 | Status flips over 2–100 launches (default 30) | Widget only; 4 vCPU / 8 GB / 300 GB floor |
| CTRF github-test-reporter | MIT | Flake rate over ≤100 previous runs | GitHub Actions only; history is artifact retention |
All three are real. All three are shallow in the same way, which is the actual story.
Allure: three products, one name
This is the most-confused thing in the category, so state it precisely.
Allure Report 2 — Apache-2.0, free, still actively released (v2.46.1 in September 2026). It was not abandoned when v3 arrived.
Allure Report 3 — also Apache-2.0 and free. v3.0.0 shipped in December 2025 after 28 betas; anything describing it as beta is out of date. Rewritten in TypeScript with a plugin system and real-time watch reporting.
Allure TestOps — proprietary, from Qameta Software, priced per user per month. Not open source in any sense.
They share a name and a lineage, not a licence — and their flaky algorithms differ. The free tool flags a test when it “got the Failed status at some point within the latest 5 launches, the test got the Passed status at least one time since then, and the test got the Failed or Broken status in the latest launch.” TestOps uses a stricter rule: at least three status transitions within the ten most recent executions.
The catch in the free version is history plumbing. Allure 2 requires you to carry it yourself — “copying the history subdirectory into the test results directory before generating the next test report” — and retains “data from up to 20 latest reports.” Allure 3 replaces that with an append-only history file, which is a real improvement.
Twenty reports is roughly a day of CI on an active repository. It is enough to spot a flake; it is not enough to answer whether a test has been degrading since April.
ReportPortal: maintained, and tightening
There is a rumour that ReportPortal has gone quiet. It has not. Commits land daily across its services and the latest release shipped in August 2026.
What probably started it was an April 2026 post, Refocusing ReportPortal, which announced that from v26.1 there is no out-of-the-box Active Directory integration, and that “all SSO (SAML) and LDAP functionality will move into ReportPortal’s paid subscription.” GitHub authentication stays free. The same post says: “ReportPortal remains open source. Our mission remains the same.”
That is an open-core tightening — worth knowing if you were relying on LDAP, irrelevant otherwise.
Two things to get right about its capabilities, because they are routinely merged:
- Auto-Analysis is defect triage, not flakiness. It performs “automated defect triaging and defines the reason for the test item failure.” The documentation never mentions flakiness.
- Flaky detection is a separate, deterministic dashboard widget — the Flaky Test Cases table, which lists a case “if test case status has changed at least once from Passed to Failed or from Failed to Passed in the specified previous launches.” No ML involved.
The real cost is infrastructure. The documented minimum is 4 vCPUs, 8 GB RAM and 300 GB of SSD — and that is the floor, not a recommendation. This is a self-hosted platform with PostgreSQL, OpenSearch, RabbitMQ and MinIO behind it. “Free” here means no licence fee, not no cost.
CTRF: the interesting outlier
CTRF is a JSON schema rather than a tool, but its github-test-reporter (MIT) does something genuinely clever: it computes flake rates across runs without a database.
The mechanism is to write the CTRF report to a GitHub Actions artifact, then re-download previous artifacts on the next run and recompute. It will look back across up to 100 previous runs.
Be precise about what that is. It does not store history — it borrows GitHub’s artifact retention, which defaults to 90 days, and recalculates every time. That is an elegant hack with a hard ceiling, and it only works inside GitHub Actions.
The npm ctrf package pulls close to 700,000 downloads a week, so the format has real reach. The Jenkins plugin does not: ctrf-json has around 15 installs and its last release was July 2024. Treat it as dormant.
The number that says the most
Jenkins publishes install counts, which makes it the only place you can see adoption rather than infer it. As of the mid-2025 data:
| Plugin | Installs |
|---|---|
| junit (ingest and display results) | 257,133 — 96.6% of ~266,000 controllers |
| htmlpublisher | 91,000 |
| allure-jenkins-plugin | 15,858 |
| flaky-test-handler | 1,124 — 0.43% |
A 225:1 ratio between ingesting test results and doing anything historical with them.
Nearly every Jenkins installation on earth parses JUnit XML — which is ingestion, not test observability. Fewer than half a percent have installed anything that reasons about flakiness across builds. That is not a tooling gap — the tooling exists and is free — it is an awareness gap, and it is the most interesting number in this post.
One caveat: Jenkins’ published statistics currently run to mid-2025, so date-stamp anything you quote from them.
What is not open source, despite appearances
A pattern worth recognising, because it catches people:
Trunk and BuildPulse both publish MIT-licensed uploader CLIs. The flaky detection happens in a proprietary hosted service. Qase open-sources its reporters and qasectl under Apache-2.0 while the platform stays commercial.
None of these are dishonest — the uploaders genuinely are MIT. But an open-source uploader tells you nothing about the licence of what it uploads to, and a GitHub repository with an OSI badge is not evidence that the product is open source.
Also worth noting so you do not go looking: GitLab ships no flaky-test detection. Unit test reports are available on Free, Premium and Ultimate, but there is no flaky-test feature at any tier. Datadog, Develocity, Cypress Cloud, Currents, Buildkite Test Engine and Allure TestOps are all commercial.
And two more that are less open than they look: Testkube is a hybrid, licensed “variously under the Testkube Community License (TCL) and the MIT license” — source-available, not OSI. sorry-cypress is MIT but effectively stale, with its last release in January 2024.
The DIY path is not a path
The natural engineer’s instinct is to skip all of this: push test results to Prometheus, chart them in Grafana, done.
Prometheus’ own maintainers advise against it, in writing. A CI job is short-lived, so the only route is the Pushgateway, and its documentation says “the only valid use case for the Pushgateway is for capturing the outcome of a service-level batch job” and that it “never forgets series pushed to it and will expose them to Prometheus forever unless those series are manually deleted.” The project’s stated non-goals include, flatly: “The Pushgateway is not an event store.”
A test run is an event. There is no TTL — the feature was rejected as an anti-pattern.
The dashboard ecosystem agrees by omission. Across roughly ten thousand published Grafana dashboards, searching for junit, pytest or cypress returns none. Every “test results” dashboard is load-testing or InfluxDB-backed. Grafana Labs publishes official k6 dashboards for performance data and nothing equivalent for functional test results.
You can build this. It is genuinely a few weeks of work plus indefinite maintenance of a schema nobody else uses.
An honest recommendation
If you want free and you are on GitHub Actions: CTRF’s github-test-reporter. Cheapest possible path to a real flake rate, no infrastructure, and it will look back 100 runs.
If you want free and self-hosted: Allure Report 3, and actually configure history — most teams never do, which is why they conclude Allure “doesn’t do flaky detection” when it does.
If you want more and can run infrastructure: ReportPortal, budgeting for the 4 vCPU / 8 GB / 300 GB floor and the operational work that comes with a four-service stack.
Where the open-source path genuinely stops is depth of history. Allure caps at 20 reports, ReportPortal’s widget looks back at most 100 launches, CTRF borrows a 90-day artifact window. Each is enough to answer “is this test flaky right now” and none is enough for “has this test been getting worse since spring,” which is the question that actually informs a reliability sprint.
Our own position, stated plainly: Qualflare is commercial and it sits exactly in that gap — durable storage of results across runs, with flaky scoring and failure clustering on top. That is not a claim the OSS options are bad; it is a claim about where they end, and the honest advice for most teams is to start with Allure or the CTRF reporter, cost nothing, and only look further when the twenty-report ceiling is the thing actually blocking you. A comparison of what each result format can carry is in test report formats compared, and the general problem is in what is test observability.
Frequently asked questions
Is there an open source tool that detects flaky tests?
Yes, three. Allure Report flags a test as flaky when it failed within the last five launches, passed at least once since, and is failing in the latest one. ReportPortal ships a Flaky Test Cases widget counting status flips over a configurable window of 2 to 100 launches, defaulting to 30. CTRF’s github-test-reporter computes a flake rate across previous GitHub Actions runs. All three are OSI-licensed and self-hostable.
Is ReportPortal still maintained?
Yes. Commits land daily across its services and the latest release, 26.0.5, shipped in August 2026. The confusion comes from an April 2026 post titled “Refocusing ReportPortal”, which announced that SSO/SAML and LDAP move to the paid subscription and that out-of-the-box Active Directory support ends in v26.1. That is an open-core tightening, and the same post states plainly that ReportPortal remains open source.
What is the difference between Allure Report and Allure TestOps?
Allure Report is Apache-2.0 and free — both the long-standing v2 and v3, which shipped in December 2025. Allure TestOps is a separate, proprietary commercial product from Qameta Software, priced per user per month. They share a name and a lineage, not a licence. Their flaky-detection algorithms also differ: TestOps requires at least three status transitions in the ten most recent executions.
Can you build test observability on Prometheus and Grafana?
Not on a supported path. Prometheus’ own documentation states that the only valid use case for the Pushgateway — the only route for a short-lived CI job — is capturing the outcome of a service-level batch job, that it never forgets pushed series, and explicitly that it is not an event store. A test run is an event. Of roughly ten thousand published Grafana dashboards, searches for junit, pytest and cypress return none.
Are Trunk or BuildPulse open source?
Their uploaders are; the products are not. Both publish MIT-licensed command-line tools that send results to a proprietary hosted service where the flaky detection actually happens. This is the same pattern as Qase open-sourcing its reporters under Apache-2.0 while the platform stays commercial. An MIT uploader tells you nothing about the licence of the thing it uploads to.
