RSpec Flaky Tests: --bisect, --seed, let vs let!

RSpec has the best tooling for order-dependent failures of any test framework, and most Ruby teams never use it. --bisect will take “this suite fails maybe one run in eight” and hand you back a specific two-example command that fails every time. That is the entire problem, solved, in one flag.

The catch is that order dependence is only one of RSpec’s flakiness sources, and the others are cultural rather than technical: before(:context), shared database state, and a let! habit that quietly changes what your tests do.

Where RSpec flakiness actually comes from

Source	Why it flakes	What finds it
before(:context)	Runs outside the per-example transaction, so its data is never rolled back	--bisect
let! at group scope	Creates records for every example, inflating counts an unrelated .first depends on	--bisect
Shared class/global state	Leaks between examples; which one breaks depends on order	--order rand
Non-transactional system tests	Browser runs on a separate connection and cannot see uncommitted data	DatabaseCleaner config
Time and timezone	Time.now near a boundary, or a frozen clock that never got unfrozen	Repetition

The first two account for most of it, and both are order-dependent — which is exactly what --bisect was built for.

--bisect: the flag worth knowing

rspec --seed 41234 --bisect

It “repeatedly runs subsets of your suite in order to isolate the minimal set of examples that reproduce the same failures.” Give it a seed that fails and it narrows to the smallest reproduction, then prints a command you can run directly.

Two practical details. --bisect=verbose shows each subset it tries, which is worth using the first time so you trust what it is doing. And Ctrl-C aborts but still reports the most minimal reproduction found so far — on a large suite you can stop it early and usually still get something useful.

There is a bisect_runner setting with :fork (the default) and :shell modes. If bisection behaves oddly in an app with heavy boot-time state, :shell is the slower but more isolated option.

--seed and reproducing an order

rspec --order rand        # randomise, print the seed
rspec --seed 123          # identical to --order rand:123
rspec --order defined     # source order (the default)

--seed 123 and --order rand:123 are the same thing. RSpec prints the seed on every randomised run, which is what makes a CI failure reproducible: take the seed from the failing build, run it locally, get the same order.

There is also --order recently-modified, which is genuinely useful for a fast feedback loop but should never be your CI ordering — you want CI randomised so order dependence surfaces.

If you are not running randomised in CI, you have order dependence and do not know it yet. That is not a prediction about your suite specifically; it is what happens to any suite that has never been shuffled.

--only-failures and --next-failure

# spec/spec_helper.rb
RSpec.configure do |c|
  c.example_status_persistence_file_path = "tmp/examples.txt"
end

With that set, --only-failures reruns just the previously failing examples, and --next-failure is shorthand for --only-failures --fail-fast --order defined.

Without the setting, RSpec aborts with a clear message telling you to configure it. Worth noting these filter, they do not retry — the distinction matters, because rerunning a failure to see if it passes is not the same as RSpec deciding a passing retry counts.

let vs let!, precisely

This is the most misunderstood pair in RSpec, and the source settles it:

def let!(name, &block)
  let(name, &block)
  before { __send__(name) }
end

let! is literally let plus a before hook that calls it. So:

• let is lazy. The block runs the first time the name is referenced in an example, then the value is memoized for that example.
• let! forces evaluation before every example in scope, referenced or not.

Memoization is per-example, not per-group. RSpec builds a fresh memoization store in initialize and instantiates a new example-group object for each example, so two examples never share a let value.

The flakiness angle is that let! creates records for every example in its scope, including examples that do not care. That inflates your database state, makes tests sensitive to record counts and ordering, and turns an unrelated .first or .last into a coin flip. A let! at the top of a large describe block is a common way to make a suite mysteriously order-dependent.

before(:context) is the prime suspect

If you have one flaky RSpec suite and no idea where to start, look here first.

before(:context) — also spelled before(:all) — runs before the per-example transaction opens. rspec-rails documents the consequence directly: “before(:context) hooks are invoked before the transaction is opened… If you don’t do that, you’ll leave data lying around that will eventually interfere with other examples.”

That is the whole failure mode. Records created there survive every rollback, accumulate across the run, and break whichever example happens to run after them — which changes with ordering, which is why it presents as flakiness rather than a straightforward failure.

RSpec takes this seriously enough to raise if you try to use let or subject inside before(:context), explaining that they “exist to define state that is reset between each example, while before(:context) exists to define state that is shared across examples.”

rspec-rails also warns about a subtler version: an object held across the transaction boundary does not know about the rollback, so “the object and its backing data can easily get out of sync.”

The fix is nearly always to move the setup to before(:each) and accept the cost, or to build the data in a way that does not touch the database.

Transactional fixtures and DatabaseCleaner

config.use_transactional_fixtures = true (aliased as use_transactional_examples) is the generator default and wraps each example in a transaction that is rolled back afterwards. It is the right default and handles most isolation.

The official escape hatch, in rspec-rails’ own words, is “If you prefer to manage the data yourself, or using another tool like database_cleaner… config.use_transactional_fixtures = false.” The usual reason is a JavaScript-driver system test where the browser runs in a separate connection that cannot see uncommitted data.

One currency note: DatabaseCleaner split into per-ORM gems. Use database_cleaner-active_record, not the old meta-gem.

Getting results into CI

rspec --format progress \
      --format RspecJunitFormatter --out tmp/rspec.xml

rspec_junit_formatter is the standard route, and you can combine formatters so you keep readable console output alongside the XML. It is maintained but slow-moving — 0.6.0 dates from 2022, with repository activity since.

What comes out is ordinary JUnit XML, with all the ambiguity that carries — including no way to express a retry, which matters here more than most because of the next section.

RSpec has no retry, and the popular gem is abandoned

RSpec core has no retry mechanism. A pull request proposing one was closed unmerged, with contributors directed to build it externally.

The gem everyone recommends, rspec-retry, is archived, and its last release was 0.6.2 in November 2019. That is not “mature and stable” — it predates several RSpec releases. If you are adding it to a new project in 2026 on the strength of a blog post, check that date first.

This is arguably fine. Retries hide flakiness rather than fixing it, and RSpec giving you --bisect instead of a retry flag is a defensible opinion about which of those is the better default. But it does mean that if you want to know which of your specs are unreliable, RSpec will not tell you — --bisect finds a reproducible ordering bug, and genuine non-determinism that fails one run in fifty regardless of order is a different problem it cannot narrow.

A CI setup

- name: Specs
  run: |
    bundle exec rspec \
      --order rand \
      --format progress \
      --format RspecJunitFormatter --out tmp/rspec.xml

- name: Upload results
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: rspec-results
    path: tmp/rspec.xml

Randomised order, JUnit XML out, uploaded even on failure. When a build fails on ordering, take the printed seed and run rspec --seed <seed> --bisect locally — that is the whole loop, and it works.

The obligatory disclosure: Qualflare is ours. It reads RSpec output and scores flaky specs from history across runs. It does not run your specs. It also does not replace --bisect, which is better at what it does than anything a dashboard can offer — if your failures are order-dependent, bisect first. History only earns its keep for the failures bisect cannot pin down, the shape covered in the complete guide to flaky tests.

Frequently asked questions

How do you find order-dependent RSpec failures?

Use rspec --bisect. It repeatedly runs subsets of the suite to isolate the minimal set of examples that reproduce the same failures, so instead of “this fails sometimes” you get a specific short command that fails reliably. Use --bisect=verbose to watch what it is trying, and note that pressing Ctrl-C aborts and still reports the most minimal reproduction found so far.

What is the difference between let and let! in RSpec?

let is lazy — the block runs the first time you reference the name in an example, then the value is memoized for that example. let! is defined as let plus a before hook that immediately calls it, so the value is created before every example whether you reference it or not. Memoization is per-example, not per-group, because RSpec instantiates a fresh example group object for each example.

Why does before(:context) cause flaky RSpec tests?

Because it runs before the per-example transaction opens, so anything it creates in the database is not rolled back and leaks into later examples. rspec-rails documents this directly, warning that you will otherwise leave data lying around that eventually interferes with other examples. RSpec also refuses to let you call let or subject inside before(:context) for the same reason.

How do you reproduce a specific RSpec test order?

Run with the seed from the failing run — rspec --seed 123, which is identical to --order rand:123. RSpec prints the seed it used on every randomised run, so a CI failure can be replayed locally in the exact same order. Ordering options are defined (the default), rand, rand:SEED, and recently-modified.

Does RSpec have a built-in retry?

No. RSpec core has no retry mechanism, and a pull request proposing one was closed unmerged with contributors directed to build it externally. The commonly-suggested rspec-retry gem is archived and its last release was 0.6.2 in November 2019, so treat any advice recommending it as stale. --only-failures filters to previously failing examples but does not retry.