Product news and testing tips.
Pass/fail is one bit of information — not enough to trust a mobile test suite. A real mobile test report tracks a test's flakiness rate over time, not just its latest result; which devices and OS versions its failures cluster on; whether its duration is drifting upward; whether it failed by crashing, by a bad assertion, by timing out, or because CI infrastructure broke; how many retries or self-heals it took to go green; how much evidence — screenshot, video, log — is attached to a failure; and how the same logical test behaves on Android versus iOS. Qualflare surfaces most of this from Espresso, XCTest, and Maestro results, but it doesn't control device provisioning, so device/OS-level breakdown stays an honest gap.
Key takeaways
- A single pass/fail result can't show whether a test is trustworthy — flakiness rate has to come from run history, not one build.
- Mobile build flakiness rose from 10% of teams in 2022 to 26% in 2025, per Bitrise's analysis of 10M+ builds — and teams that track it saw roughly 25% fewer flaky reruns.
- Google Play itself treats aggregate crash rate (1.09% bad-behavior threshold) and per-device-model crash rate (8% threshold) as different signals — the same logic applies to test failures clustering on specific devices or OS versions.
- Failure type matters: a crash, an assertion failure, a timeout, and a CI infrastructure failure point triage in different directions, even though bare JUnit-XML calls all four 'FAIL.'
- Evidence quality varies by framework: Maestro captures screenshots and logs automatically but stores them separately from its JUnit-XML report, and the standard XCTest-to-JUnit-XML converter (a7ex/xcresultparser) doesn't carry .xcresult screenshots through at all.
- Qualflare surfaces AI failure clustering, historical flaky scoring, and per-launch release risk from mobile results — but it doesn't compute device/OS-level breakdowns, since it doesn't control device provisioning.
A green Espresso, XCTest, or Maestro run tells you exactly one thing: this specific run passed. It doesn’t tell you whether the same test failed twice last week, whether it’s only breaking on one device model, whether it’s taking longer to finish than it did a month ago, or what kind of failure it produced the last time it went red. Pass/fail is a single bit of information — enough to gate a build, not enough to trust a suite. This post covers what mobile teams should track instead: flakiness trend, device/OS fragmentation, duration drift, failure type, retry/self-healing counts, evidence quality, and cross-platform comparison.
Worth saying early: Qualflare is a results and observability layer, not a device-execution cloud — it ingests whatever JUnit-XML-compatible file Espresso, XCTest, or Maestro already produced, wherever the tests ran. That framing matters for this post specifically, because it draws a real line between what a results layer can and can’t surface — a line this post won’t blur.
What a green build doesn’t tell you
| Metric | What it tells you | Why pass/fail misses it |
|---|---|---|
| Flakiness rate | Whether a test is trustworthy, not just what it did once | A single run can’t distinguish a flake from a genuine pass |
| Device/OS clustering | Whether a failure is app-wide or device-specific | Pass/fail carries no device or OS context |
| Duration trend | Whether a test is quietly degrading before it fails | A passing test with no duration history looks identical to a healthy one |
| Failure type | Crash vs. assertion vs. timeout vs. infrastructure | JUnit-XML calls all four FAIL with no distinction |
| Retry/self-heal count | Whether a “pass” was actually stable or rescued | A retried-then-passed test reports the same as a first-try pass |
| Evidence attached | Whether a failure is actually debuggable | A bare stack trace and a screenshot-plus-video tell very different stories |
| Cross-platform delta | Whether the same logical test behaves differently on Android vs. iOS | Each platform’s report is siloed unless something aggregates them |
Seven signals a mobile pass/fail count hides
Flakiness rate per test, not just its current status
A test that passed today and failed yesterday on the exact same code is flaky — but you can only see that by looking at its history across runs, not at today’s result in isolation. This matters more in mobile than it used to: Bitrise’s Mobile Insights Report 2025 analyzed over 10 million builds and found the share of teams experiencing any test flakiness rose from 10% in 2022 to 26% in 2025 — and the same report found teams that actively track flakiness saw roughly 25% fewer flaky reruns. A pass/fail count with no memory can’t participate in that improvement; it has nowhere to store the pattern. Read more on this site’s guide to flaky mobile tests for the platform-specific causes behind the trend.
Device and OS fragmentation: where a failure clusters, not just that it happened
A failure that only happens on one device model or one OS version is a different problem than a failure that happens everywhere — and treating them the same wastes triage time. Google itself draws this exact distinction for crash reporting: Android Vitals’ “bad behavior” thresholds flag an app at 1.09% user-perceived crash rate in aggregate, but at a much lower bar of 8% crash rate on a single device model — because a defect concentrated on one device is a real, actionable signal even when the aggregate number still looks fine. The same logic applies to test failures: a test failing only on Android 12 emulators, or only on one simulator runtime, is telling you something specific that a flat pass/fail count erases.
Duration trends: a test getting slower is a leading indicator
A test that’s still green but measurably slower than it was a few weeks ago is often an early warning, not a non-issue. Growing test data, a slow dependency, resource contention on shared CI runners, or a genuine app-side performance regression can all show up as duration drift long before the test actually starts failing outright. Tracking duration as a trend — this run compared to the last twenty, not just today’s number — catches problems while they’re still cheap to fix.
Failure type: crash vs. assertion vs. timeout vs. infrastructure
A test that crashes the app, a test that fails an assertion, a test that times out waiting for an element, and a test that fails because the CI runner ran out of disk space all report as the identical FAIL in bare JUnit-XML — but they point triage in four different directions. A crash usually means an app-side regression; an assertion failure means the expected state genuinely changed; a timeout often means a synchronization or environment problem (see this site’s Espresso and XCTest flaky-test posts for framework-specific timing causes); and an infrastructure failure means the test itself may be fine. Reporting that can’t separate these forces a human to open every failure just to sort it into a bucket a good report would have sorted automatically.
Retry and self-healing counts, when your framework produces them
If your test runner retries failed tests or a self-healing tool patches a broken locator mid-run, a test can go green while having genuinely struggled to get there. That struggle is real signal — a test that needs two retries every third run is not as stable as one that’s never needed a retry, even though both currently show 100% pass rate. This is execution-layer behavior, not something a results layer performs itself: Qualflare doesn’t run tests, retry them, or self-heal locators. What it can do is read the downstream effect in run history — a test repeatedly rescued shows up as unstable in its historical flaky score, even without a dedicated retry-count field.
Evidence quality: what’s attached to a failure, not just that it happened
A failure with a bare stack trace and a failure with a screenshot, a short video, and a device log attached are not equally debuggable, even if the underlying bug is identical. This varies by framework in ways worth knowing before you assume evidence just travels with the result. Maestro’s own docs confirm it captures screenshots and device logs automatically on failure — but stores them in a separate output directory from the JUnit-XML report itself, not bundled together. On iOS, Xcode attaches screenshots to a .xcresult bundle automatically on UI test failure, but a7ex/xcresultparser — the actively maintained tool for converting .xcresult to JUnit-XML — documents extracting pass/fail status, duration, and coverage data, with no mention of carrying screenshot attachments through. In both cases, the standard path to a portable JUnit-XML report leaves visual evidence behind unless a team deliberately preserves and links it separately.
Per-platform comparison: same logical test, Android vs. iOS
Teams running the same user flow as both an Espresso test and an XCUITest test rarely think to compare how it performs on each platform side by side — because the two reports usually live in entirely separate places. A flow that’s rock-solid on iOS but flaky on Android (or the reverse) is a genuinely useful signal about where a team’s platform-specific risk actually sits, and it only surfaces when both platforms’ results land in one dashboard instead of two siloed ones.
What Qualflare actually surfaces — and what it honestly doesn’t
From Espresso, XCTest, and Maestro results, Qualflare’s AI applies failure clustering (grouping failures that share a root cause), historical flaky scoring (from pass/fail patterns across runs, not one result), and per-launch release risk — the flakiness-rate, failure-type, and cross-platform-comparison signals above are exactly what that analysis is built to surface. What it honestly doesn’t do: compute a device- or OS-model breakdown on its own, because Qualflare never provisions or touches a device — it has no independent visibility into which emulator, simulator, or real device produced a given result unless that metadata was already present in the file it ingested. If your CI job or device farm tags results with device/OS information, it can travel through; if the input never had it, no results layer can report a breakdown that doesn’t exist in the data. Read more on what mobile test observability actually covers, and where its boundary sits.
Start free with Qualflare — send Espresso, XCTest, or Maestro results and see flakiness trend, failure type, and cross-platform comparison instead of a single pass/fail count.
Frequently asked questions
Why isn’t pass/fail enough for mobile test reporting?
Because pass/fail is a single bit of information about one run. It can’t show whether a test is flaky, whether it’s slowing down, what type of failure occurred (crash, assertion, timeout, or infrastructure), how many retries it took to go green, or whether it’s failing only on specific devices or OS versions. All of that history sits behind the same reported PASS or FAIL.
What’s the difference between a flaky mobile test and a genuinely broken one?
A flaky test passes and fails on the same, unchanged app build across different runs; a broken test fails consistently after a real regression. Telling them apart requires historical pass/fail data across many runs, not a single result — which is why flakiness rate, tracked over time, is the first metric worth adding beyond pass/fail.
Can Qualflare break failures down by device model or OS version?
Only if that data is present in the JUnit-XML Qualflare ingests — Qualflare doesn’t provision or control devices, so it has no independent visibility into which physical device or emulator a test ran on. If your CI job or device farm tags results with device/OS metadata, that data can travel through with them; if it doesn’t, Qualflare can’t surface a breakdown that was never in the input. This is a real limit, not a feature we’ve built.
What counts as good evidence on a mobile test failure?
At minimum, a stack trace or error message; ideally also a screenshot at the point of failure, a short video of the steps leading up to it, and the relevant device or emulator log. Xcode attaches screenshots to .xcresult automatically on UI test failure, and Maestro writes screenshots and device logs to its own output folder — but most JUnit-XML conversion paths, including the standard XCTest-to-JUnit-XML converter, don’t carry those attachments through, so evidence has to be preserved and linked separately if you want it available at triage time.
Does test duration matter if the test is still passing?
Yes. A test that’s still green but measurably slower than it was a few weeks ago is often an early signal of a real problem — a slow dependency, growing test data, resource contention, or an app-side performance regression — showing up before the test actually starts failing. Tracking duration as a trend, not a per-run number, catches that early.
Does Qualflare track retries or self-healing on mobile tests?
Qualflare doesn’t run tests, retry them, or perform self-healing itself — that’s execution-layer behavior owned by your test runner or a self-healing tool. What Qualflare does track is the downstream effect: a test repeatedly rescued by retries or a healed locator shows up as unstable in its historical flaky score across runs, even without a dedicated retry-count field.
Sources
Related Posts
