Test Automation ROI: Why the Standard Formula Is Wrong

Every vendor publishes the same equation:

ROI = (Benefits − Costs) / Costs

BrowserStack, SmartBear, Ranorex, Katalon and Perfecto all give a variant of it. The formula is arithmetic and cannot be wrong. What goes into it usually is — and the most consequential omission has been documented since 1997, which makes the persistence of the standard model the interesting part.

The assumption that breaks it

The benefits term is nearly always some form of (manual time − automated time) × tests × runs. Buried in that is an assumption: once written, a test runs indefinitely at near-zero cost.

Douglas Hoffman measured that assumption in 1999 and reported the result in Cost Benefits Analysis of Test Automation:

“we computed the average number of automated test runs before maintenance and found it to be 1.2; four times out of five the tests only worked once before we had to redo them.”

1.2 runs. If that number is even directionally right for your suite, a model that amortises build cost over hundreds of runs is not slightly optimistic — it is describing a different activity.

Hoffman writes the standard formula out formally as his equation (1) and then dismantles it, noting it “does not account for overhead or maintenance costs associated with automation.” That sentence is from 1999 and remains an accurate description of most calculators shipping today.

Cem Kaner had already supplied the other half in Improving the Maintainability of Automated Test Suites (1997):

“It usually takes between 3 and 10 times as long… to create, verify, and minimally document the automated test as it takes to create and run the test once by hand.”

Put the two together and the arithmetic gets uncomfortable. If automating costs 3–10× a manual run and the test averages 1.2 runs before needing work, you are not obviously ahead.

That is not an argument against automation — a suite that catches one production defect can pay for a great deal of maintenance. It is an argument that the standard formula measures the wrong thing.

The critique is peer-reviewed, and old

This is not a blogger’s contrarianism. Ramler and Wolfmaier, Economic perspectives in test automation: balancing automated and manual testing with opportunity cost, AST 2006, state it directly in the abstract: they “reveal problems in the overly simplistic cost models commonly used to make decisions about automating testing” and propose an opportunity-cost model instead.

James Bach’s Test Automation Snake Oil (1999) makes the same point in plainer language: “Most analyses of the cost of test automation completely ignore the special new tasks that must be done just because of the automation.”

Twenty-seven years later, Michael Bolton reverse-engineered a modern vendor calculator and reported: “Opaque formulas; undefined and unexplained terminology; unnamed and inexplicable constants; weird, inexplicable coefficients in the calculation; poor testability; hilarious bugs sitting right there on the surface.”

What the calculators actually compute

Which is checkable, so I checked.

Vendor	Maintenance as a cost	Rerun / flake term
BrowserStack (guide)	✅ with a sub-formula	✅
Ranorex, Katalon (blogs)	✅	❌
SmartBear (ebook)	◐ folded into a constant	❌
Tricentis (calculator)	❌	❌
Katalon (calculator)	⚠️ modelled as a saving	❌
Sauce Labs (calculator)	⚠️ modelled as a saving	❌

Credit where it is due: BrowserStack’s guide is the exception and it undercuts the easy version of this argument. It publishes an explicit maintenance formula — “maintenance time for one failed test case × % of failed tests per test run × number of test cases × number of test runs” — and lists “Ignoring Maintenance Costs” and “Neglecting Test Failures and Flakiness” among its own named pitfalls. It is the only vendor guide I found with a flake term at all.

The calculators are worse than the guides. Sauce Labs’ is a white-labelled third-party widget whose arithmetic is readable in the page’s own JavaScript, and it has three problems worth naming:

• The maintenance input — “What percent of the team’s time is spent maintaining and upgrading your test infrastructure?” — is multiplied by payroll and by 0.90 and added to savings. The model assumes 90% of maintenance disappears and none is incurred.
• No licence cost is subtracted anywhere. It collects a $300,000 licensing figure, displays it, and never uses it in any total.
• The output is labelled “Total Savings over Three Years” on a page titled ROI Calculator. Savings and return are not the same quantity.

The template also still contains another client’s form fields in its HTML, asking how long it took to deploy HashiCorp Vault. Katalon’s coefficients, by contrast, are computed server-side and cannot be inspected at all — you are asked to trust “actual data from our customers and assumptions from industry benchmarks.”

One more, offered without comment: Ranorex’s published worked example gives benefits of $90,000 and costs of $45,000, applies its own formula, and prints 200%. Its own formula yields 100%.

The number nobody has

The obvious fix is to add a maintenance term. The obvious question is what coefficient to use — and there is no published answer.

No survey or study establishes a general share of automation effort spent on maintenance. The percentages in circulation are untraceable; the academic literature concedes the gap directly, noting that “empirical data on maintenance costs from real, industrial systems are limited.”

The best real data is narrow and should be quoted as such. Alégroth, Feldt and Kolström (Information and Software Technology, 2016) studied Visual GUI Testing in industry and found maintenance was 61.5% of development cost across one system transition and 17.5% across another. One interviewee reported that “maintenance could take up to 60 percent of the time spent on JAutomate testing each week.” That 60% is a single person at one company, and it includes execution time.

Ranorex — a vendor, and therefore arguing against interest — offers a range: 10–25% of total automation effort in well-maintained suites, 60–70% in brittle ones. Unsourced, but the spread is the honest part. The answer depends almost entirely on how good your suite is, which is precisely why a universal coefficient does not exist.

Flakiness is the term everyone omits

A flaky test costs on every run, forever, and none of the standard models price it. The measured costs of the human half:

• Slack: about 28 minutes to triage each failure, from their own developer survey — scoped to mobile CI, and self-reported.
• Microsoft: $9.60 average cost per test inspection, from The Art of Testing Less without Sacrificing Quality (ICSE 2015). Their simulation suggested roughly a third of inspections were avoidable for Windows and Dynamics, 21.1% for Office.
• Google: 2–16% of compute resources spent rerunning flaky tests — from John Micco’s ICST 2017 presentation, not the Testing Blog, and it is compute hours rather than budget.

And then the finding that should end the discussion about precision. Asked in 2016 how Google quantified the developer cost of flakiness, Micco replied:

“We are currently working to better analyze the cost to the developer workflows caused by test flakiness — we do not yet have anything publishable out of that effort.”

Google, with 4.2 million tests and complete telemetry, could not put a number on this. A three-decimal-place figure from a vendor calculator is not better information; it is worse information presented more confidently.

What to measure instead

Stop trying to compute a single return figure and track the things that actually move it:

1. Edits per test per quarter. Your real maintenance coefficient, specific to your suite, and cheap to derive from version control.
2. Compute spent on reruns. Directly measurable, and it is pure waste.
3. Triage time on non-real failures. Even a rough per-failure estimate, multiplied by failure count, beats omitting the term entirely.
4. Defects caught before release. The benefit side. If a suite catches nothing, execution speed is irrelevant.

Those four give you a defensible per-suite picture. They will not give you a percentage to put in a slide, which is the point — the demand for a single ROI number is what the calculators exist to satisfy, and it is the reason they are built the way they are.

Our own stake, plainly: Qualflare stores CI results and scores flaky tests from history, which is where items 2 and 3 come from — you cannot count reruns or find non-real failures without keeping runs. We also publish a flaky test cost calculator, and in fairness it has the same limitation as every calculator above: it is only as good as the inputs you give it, and it does not know your suite. Use it to frame an order of magnitude, not to produce a number for a business case. The underlying argument is in the real cost of flaky tests.

Frequently asked questions

What is the standard test automation ROI formula?

Almost every vendor publishes a variant of (Benefits − Costs) / Costs, expressed as a percentage. Benefits are usually manual execution time saved — time per manual test, minus time per automated run, times the number of tests, times the number of runs. Costs are framework build time plus scripting time, and sometimes maintenance. The formula is not wrong; the inputs people put into it usually are.

What does the standard ROI formula leave out?

Chiefly that a test keeps costing you after it is written. It has to be maintained as the application changes, it consumes compute on every run, and when it is flaky it consumes human attention too — a rising flake rate is a cost line no ROI model has a row for. The savings term also assumes the manual test would otherwise have been run every time, which is often false — many automated tests replace a manual check that was performed rarely, or never.

How much of automation effort goes to maintenance?

There is no reliable general figure, and the widely-quoted percentages are untraceable. The best real data is narrow: a 2016 study of Visual GUI Testing in industry found maintenance was 61.5% of development cost across one system transition, and one interviewee reported up to 60% of weekly testing time. Treat any confident single number for this as marketing.

Are vendor ROI calculators trustworthy?

Check what they subtract. Several model maintenance as a saving rather than a cost, and at least one major vendor’s calculator collects a licensing figure, displays it, and never uses it in any total — while labelling the output “Total Savings over Three Years” on a page titled ROI Calculator. The formulas are usually server-side and unreviewable; where they are readable, they contain unexplained hardcoded constants.

How do you actually measure automation ROI?

Measure the ongoing cost, not just the build cost. Track how often each test needs editing, how much compute reruns consume, and how much time goes into triaging failures that were not real. Then compare against the defects the suite actually caught before release. A suite that catches little and needs constant attention has negative return regardless of what a formula says.