Product news and testing tips.
`./gradlew connectedAndroidTest` writes JUnit XML with no configuration, and that file is genuinely useful — but a `<testcase>` has a name, a class, a duration and a failure message, and that is the whole of it. The screenshot of the screen that failed the assertion, the steps that got there, and whether a rerun turned the test green have nowhere to live in that format. Capturing them on Android is harder than it looks: a screenshot taken when JUnit reports the failure photographs the launcher, because `@After` has already closed the activity; sharded and orchestrated runs have no merge step; and below API 29 the results directory is deleted along with the app when the Gradle task finishes.
Key takeaways
- Espresso's JUnit XML is test-level only: one <testcase> per test method, with no field for a screenshot, a step, or a retry.
- A screenshot taken from a JUnit RunListener captures the launcher, not your app: RunAfters runs the @After methods and then rethrows, so every rule has unwound by the time a listener hears about the failure.
- JUnit 4 has no 'passed' callback — a pass is the absence of a failure — and testFailure can fire more than once for a single test, which naive reporting turns into a fabricated retry.
- A @BeforeClass that throws produces one failure carrying a suite description, and the tests it guarded emit nothing at all, so a suite can silently shrink instead of going red.
- Sharded Espresso runs have no merge step: AndroidJUnitRunner shards natively with -e numShards, but nothing in the Android toolchain combines the resulting XML files.
- Android Test Orchestrator enumerates the whole suite before running it, firing start and finish for every test without executing any — a reporter that believes those notifications produces a complete, entirely green report.
- Below API 29 Gradle passes no additionalTestOutputDir, and connectedAndroidTest uninstalls the app when it finishes, deleting anything the run wrote into the app's own directories.
Espresso reporting starts from an unusually good position. ./gradlew connectedAndroidTest writes JUnit XML with nothing to configure — no reporter to install, no converter to run — and the file lands on the host at module/build/outputs/androidTest-results/connected/, documented Android Gradle plugin behaviour. Compared with iOS, where results start life inside an .xcresult bundle, that is a gift.
What it is not is a record of what happened. A JUnit <testcase> carries a name, a class name, a duration, and for a failure an exception message and stack trace. Espresso fills exactly those fields, once per test method. A test that walked through a login screen, typed into two fields, tapped a button and then failed an assertion on the next screen arrives as one line and one stack trace.
| What a failed UI test produces | Where it lives | In the JUnit XML? |
|---|---|---|
| Test name, class, duration | <testcase> attributes | Yes |
| Failure message and stack trace | <failure> element | Yes |
| The screen at the moment of the assertion | nowhere, unless the test captures it | No field exists |
| The steps that got there | nowhere, unless the test declares them | No field exists |
| Whether a rerun turned it green | the runner’s own memory, lost at exit | No field exists |
| What the test knows about itself (owner, ticket, tags) | the test’s source code | No field exists |
| Logcat around the failure | the device’s ring buffer | No |
That gap is not Espresso’s fault, and it is not the JUnit XML format’s fault either — the format predates the problem. But it does mean that everything interesting about a failed UI test has to be captured by code running inside the test, on the device, and carried out some other way. On Android that turns out to be harder than it sounds, in four specific ways worth knowing before you build anything.
The screenshot problem: your listener photographs the launcher
The obvious place to capture a screenshot on failure is a JUnit RunListener, in testFailure. It is also the wrong place, and the failure mode is quiet enough to survive code review: every screenshot comes out as a picture of the home screen.
The reason is JUnit 4’s own statement ordering. RunAfters runs every @After method and then rethrows the exception it caught. Rules unwind before the notification goes out. So by the time any listener hears about the failure, ActivityScenarioRule has closed the activity, and the screen shows whatever is behind it.
Capture has to happen inside the activity’s lifetime, which means a TestWatcher rule declared inside the rule that launched the app:
private final ActivityScenarioRule<LoginActivity> activity =
new ActivityScenarioRule<>(LoginActivity.class);
@Rule
public final RuleChain rules = RuleChain.outerRule(activity).around(new QualflareRule());JUnit runs the outermost rule first and closes it last, so the ordering in that chain is load-bearing. Reversed — outerRule(qualflare).around(activity) — the activity is already gone and you are back to photographing the launcher. An @After that closes the activity itself defeats it too, because @After runs inside all rules.
One more thing that only shows up on a device: a window with FLAG_SECURE — a payment or password screen — cannot be captured at all. UiAutomation.takeScreenshot() returns nothing. A test that fails there must not fail twice, so capture has to treat “no screenshot” as normal rather than as an error.
JUnit 4 tells you less than you think
JUnit 4’s notification shape has three edges that a reporter has to handle deliberately, because handling them naively produces a report that is confidently wrong rather than obviously broken.
There is no “passed” callback. A pass is testFinished arriving with no failure and no assumption failure before it. The verdict is inferred, never read.
testFailure can fire more than once for one test. An ErrorCollector, a @RuleChain, or an @After that throws after a failing body will each do it. Keyed naively — one case per notification — the second failure looks like a second attempt, and the report shows a retry that never happened, on a test that was never flaky.
A @BeforeClass failure has no test to attach to. It arrives as a single failure whose Description is a suite, and the tests it guarded emit nothing at all: no start, no finish, no notification of any kind. Nothing in the run says those tests were skipped. Without a synthetic case standing in for the class, the suite quietly shrinks — twelve tests one day, ten the next, and a green build both times. That is the failure mode to fear most, because a missing test looks like nothing at all.
Sharded runs have no merge step
AndroidJUnitRunner shards natively:
adb shell am instrument -w -e numShards 4 -e shardIndex 0 \
com.myapp.test/androidx.test.runner.AndroidJUnitRunnerFour shards, four emulators, four JUnit XML files — and nothing in the Android toolchain combines them. Playwright has merge-reports; Gradle has no equivalent for instrumented tests. Either upload each file to something that merges by run, or write the merge yourself and work out what to do when the same class name appears in two shards.
Android Test Orchestrator makes this more pronounced, because it runs every test in its own process. Anything written once per run is now written once per test. In our own fixture suite, eleven test methods produced eleven report files — which is fine if the filenames carry the process id, and a silent overwrite if they do not.
The orchestrator has a sharper trap, though, and we only found it because the first run of our device matrix checked the report rather than the exit code. It enumerates the suite before running it, and during that pass AndroidJUnitRunner fires testStarted and testFinished for every test without executing any of them. A reporter that believes its notifications writes a complete, plausible, entirely green report — and because the enumeration happens first, anything merging a run’s reports reads that one instead of the tests. A suite with real failures uploads as passing. The argument that marks the pass is listTestsForOrchestrator (or log for a hand-run dry run); anything reporting from inside an instrumented run has to check for it.
Getting results off the device
The JUnit XML is written by Gradle on the host, so it is always there. Anything a test writes on the device is a different matter.
androidx.test:monitor exposes PlatformTestStorage, which writes into the directory named by the additionalTestOutputDir instrumentation argument — and the Android Gradle plugin pulls that directory to build/outputs/connected_android_test_additional_output/ after the run. No adb command, no manual pull. It is how Jetpack Benchmark gets its measurements out, and it is the cleanest path available.
It also has a floor. Gradle passes additionalTestOutputDir from API 29; below that it passes nothing, and AGP says so in the build log: “additionalTestOutput is not supported on this device running API level 24.” So on API 24–28 anything you write has to go somewhere else — and every directory available to an instrumented app is app-specific: getExternalFilesDir, getExternalMediaDirs, getExternalCacheDir, getCacheDir. All four are deleted when the app is uninstalled, and connectedAndroidTest uninstalls both APKs when it finishes.
We measured this the confusing way round: the report was written correctly, ten cases and all, and by the time anything went looking, /storage/emulated/0/Android/data was empty and run-as reported the package unknown. The fix is not a different directory, because there isn’t one — it is keeping the app installed for the run you intend to collect:
./gradlew connectedAndroidTest \
-Pandroid.injected.androidTest.leaveApksInstalledAfterRun=true
adb pull /storage/emulated/0/Android/data/<your.app>/files/qualflare-results ./results
adb uninstall <your.app>Reporting a run in full
Everything above is a constraint on where a richer report can come from: inside the app process, from a rule rather than a listener, mindful of a dry-run pass, and written somewhere that survives the end of the task.
That is the shape of qualflare-espresso, which we published this week — a test-only Gradle dependency plus one instrumentation argument, reporting to Qualflare:
dependencies {
androidTestImplementation("com.qualflare:qualflare-espresso:0.1.0")
}
android {
defaultConfig {
testInstrumentationRunnerArguments["listener"] =
"com.qualflare.espresso.QualflareRunListener"
}
}It records the screenshot from inside the activity’s rule, every attempt at a test rather than the last one, steps the test declares — including from the main looper, where onActivity {} and ViewAction.perform run — and it reports nothing at all during an enumeration pass. It makes no network call: it writes files, and the CLI uploads them, which is also what lets four shards land in one launch.
The JUnit XML does not go away, and should not. It stays the path that needs no dependency, it is what your CI’s test tab reads, and for a suite where a status per test is enough, it is enough. The reporter is for the other case: when you need to know why a test failed, not that it did.
If flakiness is the specific problem, flaky Espresso tests covers the Android-specific causes — animations, idling resources, and the shared state between tests that the orchestrator exists to isolate.
