JUnit 5 testing and flaky test analysis

Writing a JUnit 5 test is the easy half. The tests themselves cause their own flakiness independent of what they test — static fields that survive JUnit's per-method instance reset, Spring contexts cached and reused across test classes, and parallel execution that turns harmless-looking shared state into a race. This guide covers both: the core annotations and parameterized tests you need to write JUnit 5 tests, and the specific reasons they turn into a flaky test — with what to configure carefully, and what to upload for real historical detection.

1. 1

   Write your first JUnit 5 test

   Annotate a test class with @Test, then @BeforeEach/@AfterEach for setup and teardown around every test. @BeforeAll/@AfterAll run once for the whole class and must be static by default — JUnit creates a new instance per test method, so a once-per-class method has no instance to run on unless the class opts into @TestInstance(Lifecycle.PER_CLASS). @DisplayName gives the test a readable name in reports.

   import org.junit.jupiter.api.*;
   
   class OrderServiceTest {
   
       private OrderService service;
   
       @BeforeEach
       void setUp() {
           service = new OrderService(); // fresh instance every test — see step 3
       }
   
       @Test
       @DisplayName("applies a 10% discount over $100")
       void appliesDiscount() {
           Order order = service.checkout(120.00);
           Assertions.assertEquals(108.00, order.total());
       }
   
       @AfterAll
       static void tearDownSuite() {
           // runs once after every test in this class — must be static
           // unless the class is annotated @TestInstance(Lifecycle.PER_CLASS)
       }
   }

2. 2

   Parameterize it and assert with intent

   @ParameterizedTest runs one test body against many inputs: @ValueSource for a single literal array, @CsvSource for multiple columns inline, @MethodSource for a static factory method when the data is more complex. Pair it with assertEquals and assertThrows, and use assertAll to run every assertion in a group and report all failures at once instead of stopping at the first. Extensions registered with @ExtendWith are how JUnit itself is extended — this is the same mechanism Mockito, Spring, and Testcontainers hook into.

   import org.junit.jupiter.params.ParameterizedTest;
   import org.junit.jupiter.params.provider.CsvSource;
   import org.junit.jupiter.params.provider.MethodSource;
   import java.util.stream.Stream;
   import static org.junit.jupiter.api.Assertions.*;
   
   class DiscountTest {
   
       @ParameterizedTest
       @CsvSource({
           "100.00, 10, 90.00",
           "50.00,  0,  50.00",
           "200.00, 25, 150.00"
       })
       void appliesPercentDiscount(double price, int percent, double expected) {
           assertEquals(expected, Discount.apply(price, percent), 0.001);
       }
   
       @ParameterizedTest
       @MethodSource("invalidPrices")
       void rejectsInvalidPrices(double price) {
           assertThrows(IllegalArgumentException.class, () -> Discount.apply(price, 10));
       }
   
       static Stream<Double> invalidPrices() {
           return Stream.of(-1.00, -50.00);
       }
   
       @Test
       void discountedOrderStaysConsistent() {
           Order order = Discount.applyToOrder(sampleOrder());
           assertAll("order invariants",
               () -> assertTrue(order.total() >= 0),
               () -> assertEquals(sampleOrder().items().size(), order.items().size()),
               () -> assertNotNull(order.appliedAt())
           );
       }
   }

3. 3

   Watch for shared static state — the classic JUnit flake

   JUnit’s default lifecycle (PER_METHOD) creates a new test instance per method, so instance fields reset automatically between tests. Static fields do not — they persist for the whole class and JVM, so a test that mutates a static list, counter, or singleton leaks state into whichever test runs next. The result passes in isolation and fails only in a certain order, under sharding, or under parallel execution: the definition of a flaky test.

   class ReportGeneratorTest {
   
       // Shared by every test in this class, and never reset.
       static List<String> generatedReports = new ArrayList<>();
   
       @Test
       void generatesQuarterlyReport() {
           generatedReports.add(ReportGenerator.create("Q1"));
           assertEquals(1, generatedReports.size()); // true alone, false once another test ran first
       }
   }

4. 4

   If you use Spring Test, watch the cached ApplicationContext too

   Spring’s TestContext framework caches an ApplicationContext keyed by configuration and reuses the same singleton beans across test classes to keep the suite fast. A test that mutates a shared bean without resetting it leaks into the next class that reuses that same cached context — order-dependent failures with the same root cause as static state, one layer up. @DirtiesContext forces a rebuild, but it is expensive; reserve it for tests that truly must dirty the context, and reset mutable singleton state in @BeforeEach where you can instead.

   @SpringBootTest
   class InventoryServiceTest {
   
       @Test
       @DirtiesContext // forces a fresh ApplicationContext for tests that run after this one
       void depletesSharedStockSingleton() {
           // mutates a bean the cached context would otherwise reuse
       }
   }

5. 5

   Reproduce intermittent failures with @RepeatedTest, then configure reruns and parallelism carefully

   @RepeatedTest(50) runs a suspect test back-to-back to force a rare race into the open — a single run tells you nothing about a test that fails once in twenty. If failures only appear once JUnit’s parallel execution is enabled, the cause is almost always shared state across threads racing on the same static field, cached bean, or file. Maven Surefire’s rerunFailingTestsCount reruns a failure within the same build and records it as flakyFailure (passed on rerun) or rerunFailure (failed every time) in the XML — useful signal, but the same rule applies here as everywhere else: a test that only passes on retry is still flaky, not fixed.

   # src/test/resources/junit-platform.properties
   junit.jupiter.execution.parallel.enabled = true
   junit.jupiter.execution.parallel.mode.default = concurrent

   <!-- pom.xml — rerun a failing test up to 2 times within the same build -->
   <plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-surefire-plugin</artifactId>
     <configuration>
       <rerunFailingTestsCount>2</rerunFailingTestsCount>
     </configuration>
   </plugin>

6. 6

   Upload JUnit XML to Qualflare for historical flaky scoring

   A single run — retried or not — still cannot prove a test is flaky; that takes pass/fail history across builds. Maven Surefire and Gradle already write JUnit XML, including any flakyFailure or rerunFailure data. Point the Qualflare CLI at it and let history do the rest: every test gets a flaky score built from its record across runs, not just the one in front of you.

For the CLI setup, the multi-module XML sweep, and the “one schema, many dialects” nuance of the format itself, see JUnit test reporting. For a framework-agnostic walkthrough of the detection setup, see how to set up flaky test detection in CI. New to the terms? See flaky test, non-determinism, test retry, and test parallelization in the glossary.

Frequently asked questions