Karate test reporting

Karate writes a readable HTML report per run and then your CI job ends and takes it with it. Qualflare turns those runs into hosted, historical reporting: every scenario as its own case, every Examples row tracked separately, tags preserved, and AI clustering of failures by root cause.

Send Karate results to Qualflare

There is no flag to add: Karate already writes a JSON report per feature as part of a normal run.

# Karate writes a JSON report per feature into target/karate-reports/
mvn test

# Or via the standalone JAR
java -jar karate.jar src/test/features/

Point the CLI at the directory and it reads every feature report in it, merging them into one launch:

# Point at the directory — one launch, however many features ran
qf my-project collect target/karate-reports/ --format karate

In CI that's one added step. Authenticate the CLI once with your Qualflare access token, stored as a CI secret — see the CLI docs.

# .github/workflows/api-tests.yml
- name: Run Karate suite
  run: mvn -B test

- name: Upload results to Qualflare
  if: always() # upload even when scenarios fail — that's the point
  run: qf my-project collect target/karate-reports/ --format karate

The if: always() line is the one people leave out. Maven exits non-zero when scenarios fail, so without it the upload runs only on green builds and your failure history — the entire point — stays empty.

Features, scenarios, and what counts as a test

Karate is Gherkin pointed at APIs rather than browsers — the Given/When/Then shape, with HTTP and JSON assertions as first-class syntax instead of step definitions you have to write:

Feature: Checkout API

  Background:
    * url baseUrl
    * header Authorization = 'Bearer ' + token

  @smoke @payments
  Scenario: rejects an expired card
    Given path 'checkout'
    And request { card: '4000000000000069' }
    When method post
    Then status 402
    And match response.code == 'card_expired'

The scenario is the test. Each one becomes a case, with its steps carried as detail underneath. That is not an arbitrary choice: a Karate scenario halts at its first failing step, so promoting steps to tests would report the cascade of skipped steps after a failure as if each were an independent failure, inflating every red number in the report. One scenario, one verdict, with the failing step named.

Scenario Outline is where the counting gets interesting. Each Examples row runs independently and arrives as its own case, carrying its example index:

  Scenario Outline: rejects <label>
    Given path 'checkout'
    And request { card: '<card>' }
    When method post
    Then status 402

    Examples:
      | label        | card                 |
      | expired card | 4000000000000069     |   # exampleIndex 0
      | declined     | 4000000000000002     |   # exampleIndex 1
      | no funds     | 4000000000009995     |   # exampleIndex 2

Three rows, three cases, three histories. If only the no funds row is intermittent, per-row history points at that specific input — whereas collapsing the outline into one case tells you only that “rejects <label> is a bit flaky”, which is not something you can act on. The same principle applies to Cucumber and to data providers in TestNG.

Tags are preserved — the feature's and the scenario's own, combined onto the case — so you can filter a report by @smoke or @payments the same way you filter a run. Worth naming because it bit us once: merging those two tag lists naively lets one scenario's tags leak onto its siblings. Tags on a case are its own plus its feature's, and nothing borrowed.

What you get on top of the HTML report

• History across runs. Karate's report describes one execution and is overwritten by the next; “has this scenario been getting flakier?” needs many.
• Examples rows as first-class cases. Flakiness attributed to the data row that causes it.
• AI failure clustering. When an expired token or a downed dependency takes out 30 scenarios, they group into one root cause rather than 30 separate failures.
• Flaky scoring from history. Karate has no scenario-level rerun, so there is no in-run attempt data — history is the only available method, and the more accurate one.
• Tags carried through. Filter and segment the report the way you segment the run.
• One launch across every suite. API scenarios next to UI and unit results instead of three disconnected pass rates.

Karate's report vs Qualflare

Complementary, and the “Partly” is honest: Karate's own HTML report is excellent for reading one run's request and response bodies in detail. Keep it for that, and add Qualflare for what one run cannot show.

Qualflare works the same with Newman, k6, CucumberJS, JUnit, TestNG and 20+ more frameworks. Prefer reading first? See Cucumber & BDD test reporting. Weighing tools? See how it compares to other test management platforms, or browse all framework reporting guides. Every reporter is open source.

Frequently asked questions