RSpec test reporting

RSpec gives you green dots, a failure list, and a seed number you will need later and did not write down. Qualflare turns those runs into hosted, historical reporting using RSpec’s own JSON formatter — no extra gem: every example as its own case, AI clustering of failures by root cause, and flaky scores that make order-dependent failures visible instead of mysterious.

RSpec’s formatters, and which one to keep

RSpec ships several formatters and — unusually — lets you run more than one at a time, each with its own destination. That is the whole integration: keep the console output you already read, and add a machine-readable file beside it.

• progress. The default green dots. Compact, fast to scan, produces no artifact.
• documentation. Nested example names as prose — the output that makes people like RSpec. Still just console text.
• json. The complete run as structured data: every example with its ID, full description, status, run time and exception. This is what to upload, and it ships with rspec-core — no gem to add.
• rspec_junit_formatter (third-party gem). JUnit XML for CI plugins that only speak that format. Strictly lossier than the JSON: the schema has nowhere to put RSpec’s pending-versus-skipped distinction.

The JSON the formatter writes looks like this:

{
  "examples": [
    {
      "id": "./spec/example_spec.rb[1:1]",
      "description": "redirects on success",
      "full_description": "Auth login redirects on success",
      "status": "failed",
      "file_path": "./spec/example_spec.rb",
      "line_number": 12,
      "run_time": 0.0412,
      "exception": { "class": "RSpec::Expectations::ExpectationNotMetError", ... }
    }
  ],
  "summary_line": "3 examples, 1 failure, 1 pending"
}

Why one JSON file isn’t reporting

Because the interesting questions in a Ruby suite are all historical ones. RSpec runs examples in random order by default, which is a genuinely good decision — it surfaces leaked state that a fixed order would hide — but it also means a failure you saw this morning may not reproduce this afternoon. One JSON file cannot tell you whether "rejects a bad password" fails one run in twenty or has been failing steadily since Tuesday, and that difference is the entire triage decision.

For history you need something that stores results over time and analyzes them.

Send RSpec results to Qualflare

Add the JSON formatter to the command you already run. Nothing in spec_helper.rb or your specs changes:

# RSpec's built-in JSON formatter, written to a file
rspec --format json --out rspec-results.json

# Keep readable console output at the same time
rspec --format progress --format json --out rspec-results.json

Then upload it:

qf my-project collect rspec-results.json --format rspec

In CI that’s one changed line and one added step. Authenticate the CLI once with your Qualflare access token, stored as a CI secret — see the CLI docs.

# .github/workflows/tests.yml
- name: Run specs
  run: bundle exec rspec --format progress --format json --out rspec-results.json

- name: Upload results to Qualflare
  if: always() # upload even when specs fail — that's the point
  run: qf my-project collect rspec-results.json --format rspec

The if: always() line is the one people leave out. rspec exits non-zero on failures, so without it the upload step never runs on exactly the builds worth recording, and your failure history stays empty.

What you get on top of the dots

• Order-dependent flakiness made visible. Random ordering turns leaked state into intermittent failures; history turns intermittent failures into a score you can act on.
• Renames keep their history. RSpec keys an example on position, not description, so rewording an it block does not orphan its past.
• AI failure clustering. When one broken factory or a missing fixture takes out 40 examples, they group into one root cause instead of 40 stack traces.
• Pending vs skipped kept apart. An example that was expected to fail and a spec someone excluded months ago are different problems, and stay different in the report.
• Per-example timing trends. run_time comes through on every case, so a slow creep is visible before it becomes a CI budget problem.
• History, trends & defects. Pass rate, slowest examples, and flakiness over time across branches.

RSpec output vs Qualflare

Complementary: keep --format documentation for local runs, add Qualflare for hosted, historical CI observability.

Qualflare works the same with pytest, Jest, Go, TestNG, Playwright and 20+ more frameworks. Chasing a specific flaky spec? See RSpec flaky tests: --bisect, --seed, and let vs let!. Weighing tools? See how it compares to other test management platforms, or browse all framework reporting guides. Every reporter is open source.

Example IDs, seeds, and the flakiness RSpec goes looking for

RSpec identifies an example by position, not name. An ID like ./spec/auth_spec.rb[1:1:2] means the second example inside the first nested block inside the first top-level group. That has a pleasant consequence and an awkward one:

# spec/auth_spec.rb
describe "Auth" do
  describe "login" do
    it "redirects on success" do ... end   # [1:1:1]
    it "rejects a bad password" do ... end # [1:1:2]  <- insert above this
  end                                      #             and it becomes [1:1:3]
end

The pleasant one: rewording an example keeps its history. In most frameworks the test name is the identity, so improving a description silently retires the old test and starts a new one with no past. RSpec is immune to that. The awkward one: inserting an example shifts every ID below it, so those examples look brand new and their history restarts. It is not data loss — the old records remain — but the continuity breaks. Where it is natural to do so, add new examples at the end of a block.

Then there is ordering. RSpec runs in random order by default and prints the seed every time:

$ rspec
Randomized with seed 41234        # different every run, by design

$ rspec --seed 41234              # reproduce that exact order
$ rspec --bisect --seed 41234     # find the minimal spec set that reproduces it

This is RSpec deliberately hunting for a bug class most suites never look for. If example A leaves a record in the database, a stubbed constant, or a memoised singleton behind, example B may pass only when it runs after A — a suite in fixed order stays green and breaks the day someone reorders a file. Random ordering converts that latent fault into an intermittent failure, which is uncomfortable and correct.

It is also why RSpec suites benefit disproportionately from history. A single random-order failure is nearly useless on its own — you cannot tell an order dependency from a genuine regression from noise. Across fifty runs the shape is obvious: an order-dependent example fails at a rate set by how often the ordering puts its partner first, while a real regression fails every time from a particular commit onward. Record the seed from a failing run, feed it to --bisect, and RSpec will narrow it to the minimal set of specs that reproduce it.

Two smaller notes. Pending and skipped are not the same thing — a skipped example never ran, a pending one ran and was expected to fail, and a pending example that unexpectedly passes is RSpec telling you the fix has landed. And --only-failures needs configuration: it depends on example_status_persistence_file_path, because RSpec has to remember the previous outcome somewhere. If that file is not persisted between CI jobs, the flag silently has nothing to work from.

Frequently asked questions