AI in Software Testing: What Actually Works

Every test tool now claims AI. Most of those claims describe one of four quite different things, and they do not all work equally well — so the useful question is never “does it have AI” but “which of the four, and how does it fail?”

This guide is the map. It is deliberately unenthusiastic in places, because the gap between what this category demonstrably does and what it is marketed as doing is wide, and the parts that genuinely work are less exciting than the parts that do not.

The four categories

Category	What it does	Maturity	Failure mode
Analysis	Clusters failures, scores flakiness, rates release risk	High	Wrong grouping — annoying, visible
Selection	Picks which tests to run from what changed	High	Skips a test that would have caught it
Maintenance	Repairs selectors when the UI changes	Medium	Silently passes a broken test
Generation	Writes test cases from code or description	Improving fast	Volume of shallow tests that all pass

The right-hand column is the one to read carefully. A tool whose mistakes are loud is safe to adopt early. A tool whose mistakes are silent needs a much higher bar, and self-healing is the category where that distinction bites hardest.

Analysis: the part that quietly works

This is where most shipping value actually is, and it is the least glamorous.

Failure clustering groups a wall of failures by likely root cause. When a shared fixture breaks and 200 tests go red, the useful output is not 200 failures — it is “one cause, 200 symptoms.” This is largely a text-similarity problem over stack traces and error messages, and it was tractable well before LLMs; language models mostly made the grouping better at handling messages that differ in wording but not in meaning.

Flaky scoring rates how likely a test is to be unreliable, derived from its pass/fail history — and a flaky test is defined by behaviour across runs, so nothing about a single run can establish it. Google’s analysis of its own corpus found almost 16% of tests showed some level of flakiness. Note what that means: it is a statistic over stored results, not a model reading your code. Any tool offering this is necessarily keeping history, and one that has just been installed cannot do it yet — a fact worth checking against any demo that shows it working instantly.

Release risk scoring rates how safe a build looks, usually combining failure counts, flake rates, coverage of changed areas, and historical outcomes. It is the most opinionated of the three and the most worth interrogating: ask what goes into the score, because a number nobody can decompose is a number nobody should gate on. That is the substance of AI quality gates — the model can advise, but the policy around it has to be explicit.

The underlying discipline also long predates the marketing: Martin Fowler’s account of eradicating non-determinism in tests sets out the same diagnosis — a test that changes result without a code change is a signal to trace, not noise to retry — a decade before anyone called it AI.

Why this category works: the ground truth is knowable. Whether two failures share a root cause, whether a test flipped outcome on unchanged code, whether a release caused an incident — these are all checkable after the fact. A model that can be scored against reality improves. A model that cannot, does not.

Selection: older than the marketing suggests

Smart test selection, also called test impact analysis, runs only the tests plausibly affected by a change. It genuinely cuts CI time substantially on large suites.

It is worth knowing that this is not new and mostly is not machine learning. The core is dependency analysis: map which tests touch which code, then intersect that with the diff. Google, Facebook and Microsoft have run variants of this at scale for well over a decade. Where modern implementations add learning, it is usually to handle the cases static analysis misses — reflection, dynamic dispatch, configuration-driven behaviour, integration tests that touch everything.

The failure mode is real but manageable: a selection that skips a test that would have caught the bug. The standard mitigation is running the full suite on a schedule — nightly, or on main — so selection speeds up the feedback loop without becoming the only thing that ever runs. This is a delivery-throughput intervention and worth measuring as one: DORA’s capability research treats fast, trustworthy feedback as what actually moves lead time.

Maintenance: the category to scrutinise

Self-healing tests detect that a selector no longer matches and repair it — falling back to a different attribute, a nearby text label, or a visual match.

The problem it solves is real. A test that breaks because a data-testid was renamed was never testing that attribute; the failure is pure maintenance cost.

The danger is specific and worth stating plainly: a heal can be wrong in a way that passes. If the test heals onto a superficially similar element — a different button with the same label, a stale element still in the DOM — it goes green while the behaviour it was supposed to protect is broken. You have converted a loud failure into a silent one, which is strictly worse than the failure you started with.

If you use self-healing, the non-negotiable requirement is an audit trail: every heal recorded, reviewable, and countable. A test that heals repeatedly is telling you something — either the selector strategy is wrong or the UI is genuinely unstable — and a rising heal rate deserves the same attention as a rising flake rate.

Generation: improving fastest, hardest to judge

AI test generation is where the most visible progress is happening, and it is genuinely useful for producing a first draft, covering mechanical variations, and filling in the obvious paths nobody wants to write by hand.

Its weakness is judgement. A model can see that a function takes an integer and generate cases for zero, negative, and maximum. It cannot know that in your domain, an order of exactly zero items is a legitimate state that must be preserved, while a negative one indicates a corrupted upstream feed and should raise rather than validate.

The failure mode is quantity masquerading as coverage. Two thousand generated tests that all pass feel like protection and may test almost nothing — they can pin the implementation’s current behaviour rather than the behaviour it is supposed to have, which means they will faithfully protect a bug. Review generated tests as you would a junior engineer’s: the mechanical parts are probably fine, the question is whether they assert anything that matters.

Agentic testing: the fifth category, and the newest

The four categories above all assume a human decides what happens and the model assists. Agentic testing is the attempt to remove that assumption — an agent that explores an application, decides what is worth testing, executes, and reports, with the loop closed rather than a human in it.

It belongs in a separate category because its failure mode is different again. The four above fail at a task you defined. An agent fails at choosing the task, which is much harder to notice: you get a report describing what it did, and nothing describes what it never thought to try. Absence of a finding is not evidence, and an agent’s coverage is far harder to audit than a written suite’s.

The honest position in 2026 is that this is the least mature category and the one moving fastest. Exploratory use — point an agent at a staging environment and see what it finds — is genuinely valuable and low-risk, because a finding is a finding regardless of how it was reached. Using an agent as your regression safety net is a different proposition, because that requires trusting what it didn’t report.

Conversational analysis is the adjacent, tamer version of the same idea: rather than an agent deciding what to test, you ask questions of results that already exist — “why did checkout fail last night?” — and the model does the retrieval and summarisation. It sits inside the analysis category rather than beside it, because the ground truth is still checkable: the answer either matches the run data or it does not. Our own take on that pattern is described in conversational test analysis.

How to evaluate a claim

Three questions separate real capabilities from marketing, and the answers get vague at exactly the point the capability thins out.

1. What technique, specifically? “AI-powered” is not an answer. “A convolutional network for visual element recognition” is. “Gradient-boosted trees over historical run data” is. A vendor that can name the method usually has one.

2. What data, and does it need mine? Anything history-derived — flaky scoring, risk rating, trend analysis — cannot work on day one. If a demo shows it working on a fresh account, the demo is using seeded data. That is fine, but you should know it, because it changes your evaluation timeline from an afternoon to a few weeks of real runs.

3. What happens when it is wrong? This is the question that matters most and gets asked least. Wrong clustering wastes a few minutes. A wrong heal ships a bug. Ask for the failure mode, not the accuracy number.

That third question is also why the honest read on this category is that analysis is safe to adopt now, maintenance needs guardrails, and generation needs review. A thorough version of this audit — checking a dozen named vendors’ claims against their own documentation — is in AI in QA: what’s real vs hype.

Testing AI systems is the other problem

Worth separating clearly, because the vocabulary collides: using AI to test software and testing software that contains AI are different problems.

The second is genuinely harder, because the thing you are testing is non-deterministic by design. A traditional assertion — this input produces this output — does not hold when the same prompt can produce different text on consecutive calls. The discipline that has grown up around it (evaluation sets, scored rather than binary assertions, judging model output with another model) borrows from testing but is not the same practice. It is non-determinism as a design property rather than a defect.

The connection back to ordinary testing is real, though, and it is history: when any single run is a sample rather than a verdict, the only meaningful signal is the distribution across many runs. That is the same reason a single test report cannot tell you whether a test is flaky.

Where the whole category rests

Every AI feature in the analysis and selection categories is downstream of one unglamorous thing: stored test results over time.

Flaky scoring is a function of history. Failure clustering is far more useful when it can say “this cluster is new” rather than just “these look similar.” Risk scoring compares this release to previous ones. Test selection needs to know which tests historically cover which code.

This is why the honest framing of most “AI testing” products is that they are data products with a model on top, and the data is the harder half. A vendor with excellent models and thirty days of your history will do less for you than a plain statistical approach with two years of it.

Full disclosure: Qualflare is our product and it lives squarely in the analysis category — failure clustering, flaky scoring from history, and per-launch risk. It does not run your tests, does not generate them, and does not heal selectors. The categorisation above is not neutral — we would say analysis is the mature part, because that is what we built. What is checkable independently is the reasoning: analysis has knowable ground truth, and self-healing’s failure mode really is silent. Judge the argument rather than the source.

Where to start

If you are adopting one thing from this category, adopt analysis, in this order:

1. Store your results first. Everything downstream needs history, and history only accumulates forwards — you cannot backfill it. This is the step teams skip and then wait weeks to benefit from.
2. Add failure clustering. It pays off immediately on any suite big enough to produce walls of red, and its mistakes are visible.
3. Let flaky scoring accumulate. After a few weeks of runs you will have a ranked list of your least trustworthy tests, which is a far better input to a reliability sprint than anyone’s recollection.
4. Only then consider selection, and keep a full run on a schedule.

Leave self-healing until you have an audit trail to put around it, and treat generation as a drafting aid rather than a coverage strategy. A survey of what specific tools do in each category is in the best AI test management tools.

Frequently asked questions

What does AI actually do in software testing today?

Four things, in descending order of maturity. It analyses results after a run — grouping similar failures by root cause, scoring which tests are flaky from their history, and rating release risk. It selects which tests to run based on what changed. It repairs broken selectors when a UI changes. And it generates test cases from code or from a description. Analysis and selection are the most reliable; generation is improving fastest.

Is AI test generation good enough to replace writing tests?

Not to replace, but genuinely useful to accelerate. Generated tests are good at covering obvious paths and mechanical variations, and weak at knowing what actually matters about your domain. The practical failure mode is volume — a large suite of shallow generated tests that all pass creates the appearance of coverage while testing very little. Review generated tests the way you would review a junior engineer’s.

Are self-healing tests a good idea?

They solve a real problem — selector churn breaking tests that were not actually testing the selector — but they carry a specific danger. If a test heals onto a different element and passes, you have converted a loud failure into a silent one, and the user-facing regression ships. Use them with an audit trail that records every heal, and treat a rising heal count as a signal that something is wrong.

How do you evaluate an AI testing claim from a vendor?

Ask three questions. What technique is it — a named model architecture, a statistical method, or unspecified “AI”? What data was it trained or fitted on, and does it need your history before it works? And what happens when it is wrong — does it fail loudly, or silently pass something? Vendors with real capabilities answer these specifically; the answers get vague exactly where the capability thins out.

Does AI in testing require storing test history?

Almost all of it does. Flaky scoring is a function of a test’s outcomes across many runs. Failure clustering is most useful when it can tell you this cluster is new versus recurring. Risk scoring compares a release to previous ones. A single test report has no history in it, so any tool offering these capabilities is necessarily storing results over time — which is worth knowing when you evaluate one.