Software Testing Statistics: Every Number, Sourced

Almost every statistic in software testing arrives without a source. Follow the ones that do have citations and a pattern emerges: a vendor page citing a vendor page citing a report that turns out to be a marketing exercise, or a document that has never been produced by anyone.

This post is what survives the check. Every number below traces to a primary document you can open, with its sample and its scope. The first section is the one that matters most, because the claim it dismantles is probably the most-repeated sentence in the field.

The 100x claim has no source

You have seen it: a defect costs 1x to fix in requirements, 10x in development, 100x in production. It appears in conference talks, vendor decks, and the business case for essentially every testing tool sold.

Follow it and it disappears.

The trail leads to Roger Pressman’s Software Engineering: A Practitioner’s Approach, which cites: [IBM81] "Implementing Software Inspections," course notes, IBM Systems Sciences Institute, IBM Corporation, 1981.

Those course notes have never been produced. Not by Pressman, not by IBM, not by anyone who has gone looking — and people have gone looking.

Worse, the institution appears to be a misnomer. The real IBM entity was the Systems Research Institute, an internal employee education centre in New York City. It was a training school. It published no studies. Glenford Myers — author of The Art of Software Testing — led a team there; Kenneth Iverson taught there. “Systems Sciences Institute” looks like a mis-transmission that has been copied ever since.

What the real research says

There is a genuine ancestor, and it is more interesting than the myth. Barry Boehm’s Software Engineering (IEEE Transactions on Computers, December 1976) contains Figure 3, “Software validation: the price of procrastination” — a log-scale chart of relative cost to fix an error by phase, drawing on IBM, GTE and a TRW survey.

Read the actual plot and the TRW medians run roughly: requirements 0.25, design 0.5, code 1.0, development test 1.9, acceptance test 5, operation about 17.

The 100 is the top of the axis, not the median. Boehm’s prose does mention 1 versus 100 man-hours, so the number is his — but his own plotted central tendency is around 17x from code, not 100x, and the chart shows a wide spread rather than a law.

Boehm then revised it himself. In Software Defect Reduction Top 10 List (IEEE Computer, 2001), he and Victor Basili added a qualifier and a correction:

“For this updated list, we have added the word ‘often’… One insight shows the cost-escalation factor for small, noncritical software systems to be more like 5:1 than 100:1.”

Three independent lines converge on ~5x

NIST measured it. The 2002 economic impacts report — the same study aggregators cite for the opposite claim — contains a table of hours to fix a bug by introduction and discovery point. Its own caption: “it was twice as costly to fix the error if it was not found until the integration phase and five times as costly if it was not detected until post-product release.” A bug found in coding took 2.4 hours; post-release, 13.1.

Boehm revised to 5:1, as above.

And the largest study on the question found no effect at all. Menzies, Nichols, Shull and Layman, Are delayed issues harder to resolve? (Empirical Software Engineering, 2017), examined 171 projects worldwide from 2006 to 2014:

“To the best of our knowledge, this is the largest study yet published on this effect. We found no evidence for the delayed issue effect.”

Fixing bugs earlier is still good practice for reasons that do not need a fabricated multiplier — shorter feedback loops, cheaper context recovery, less rework downstream. But if your business case rests on 100x, it rests on nothing.

“Developers spend 50% of their time debugging”

Also untraceable, and its actual origin is worth knowing.

It comes from a 2013 Cambridge Judge Business School news item describing a “Cambridge Venture Project” — five MBA students doing a term consulting exercise, commissioned by a company that sells a debugger. No sample size, no sampling frame, no extrapolation method, no paper. The same page reports the headline cost as $316bn and the body text as $312bn.

A second exercise by the same pairing later produced 26% — contradicting the first — and aggregator articles routinely cite both figures in the same piece.

What measurement actually shows

Finding	Source	Sample
Bugfixing 14% of workday (74 min); testing 8% (41 min)	Meyer et al., IEEE TSE 2019	5,971 responses from Microsoft professionals
A quarter of IDE time on tests — and developers “overestimated this number nearly twofold”	Beller et al., IEEE TSE 2019	2,443 engineers, IDE telemetry over 2.5 years

That second finding is the most useful sentence in this entire post, because it explains the folklore. Developers believe they spend about half their time on testing. Instrumented measurement says a quarter. The 50% figure is not invented from nothing — it is a real perception, measured, and found to be roughly double reality.

Two caveats to carry: Beller’s data is IDE-only, so it does not capture CI or command-line runs; and the same study found only 17% of projects contained tests runnable in the IDE at all.

Worth knowing where the data isn’t: the Stack Overflow Developer Survey, with roughly 49,000 respondents, asks developers nothing about testing. JetBrains’ State of Developer Ecosystem has no question about time spent testing. The two largest developer surveys in the world do not cover this, which is precisely why the space is so dependent on vendor numbers.

Flaky tests: the numbers that hold, and one that conflicts

Google’s 2016 post remains the most-cited source, and its figures check out verbatim:

• “about 1.5% of all test runs reporting a ‘flaky’ result”
• “Almost 16% of our tests have some level of flakiness associated with them”
• “about 84% of the transitions we observe from pass to fail involve a flaky test”

Those first two are different measurements and are constantly merged. 16% is tests that have ever been flaky; 1.5% is runs — a flake rate with a different denominator.

The 2017 follow-up broke it down by size across roughly 4.2 million tests in a one-week window: 0.5% of small tests, 1.6% of medium, 14% of large. Correlation with binary size r²=0.82, with RAM r²=0.76 — meaning how big a test is predicts flakiness better than which tool wrote it.

And here is a discrepancy nobody reports. Software Engineering at Google (O’Reilly, 2020) states that “our flaky rate hovers around 0.15%” — ten times lower than the 2016 blog’s 1.5%. Same company, related metric, four years apart, never reconciled. Cite either with its date; do not average them.

One more correction: the widely-quoted “2–16%” figure is compute resources spent rerunning flaky tests, not engineer time, and it comes from a 2017 ICST presentation rather than the blog.

DORA: the benchmark table you have seen is a splice

DORA is the most methodologically transparent source in this space — Bayesian structural equation modelling, published priors, credible intervals, a public errata process. It is also routinely miscited.

The current report has no benchmarks. The 2025 edition — retitled State of AI-assisted Software Development, n=4,867 — replaced the performance clusters with seven descriptive archetypes carrying no metric thresholds. The word “elite” does not appear in it.

For the familiar table you must cite DORA 2024:

Level	Change lead time	Deploy frequency	Change fail rate	Recovery
Elite	Less than one day	On demand	5%	Less than one hour
High	1 day–1 week	Daily–weekly	20%	Less than one day
Medium	1 week–1 month	Weekly–monthly	10%	Less than one day
Low	1–6 months	Monthly–6-monthly	40%	1 week–1 month

Two things to notice. Elite lead time is “less than one day,” not “under an hour” — the sub-hour figure is from 2021 and keeps getting spliced in. And the change-fail column is non-monotonic: Medium beats High. DORA addresses this and warns the ladder is not a clean ordinal ranking.

It is also five metrics now, not four — rework rate joined in 2024. If you track these, the mapping to testing is covered in DORA metrics for QA teams. And 2022 had no elite cluster at all; DORA wrote that year that “we don’t consider any cluster to be elite.” Every “DORA 2022 elite benchmark” citation is describing something that does not exist.

CI in practice: real telemetry

CircleCI publishes genuine platform data with stated samples, which makes it unusually citable:

Edition	Sample	Median duration	Main-branch success	Median recovery
2025	14,146,319 workflows	2m 43s	82.15%	63m 50s
2026	28,738,317 workflows	2.2 min	70.8%	72 min

Use the median, always: in the 2025 data the mean recovery time is 24h 15m against a median of 63 minutes — roughly 23× apart, because a long tail of never-fixed builds drags it.

The success-rate arc is worth noting rather than over-reading: 82.5% → 82.15% → 70.8%, with a partial recovery to 76.7% in a 2026 pulse. CircleCI attributes the drop to AI-generated code. That is their interpretation, not a measured cause.

The folklore section

Some numbers are not merely unsourced; they have a documented contamination chain.

“$2.41 trillion cost of poor software quality.” Quoted everywhere as a global figure; the report says US. The 2022 update is arithmetic on the 2020 number — verbatim: “Due to just inflation at 15% cumulative, our $1.56 trillion estimated in 2020, jumps to $1.8 trillion in 2022… In all the other areas of CPSQ we simply assumed that there was no growth.”

About 60% of the total comes from one assumption: that US cybercrime cost is proportional to US GDP share of a global figure published by a media company. The report’s own chart on the same page plots the FBI’s measured number for that year — $10.3 billion. The estimate is roughly 140× the measurement. And the report doubts itself in writing: “It is doubtful that all the other types of operational failures amount to just 20% of that total.”

Standish CHAOS project-failure rates. Independently examined by Eveleens and Verhoef in IEEE Software, who applied Standish’s own definitions to 5,457 forecasts across 1,211 real projects and found them “misleading, one-sided, pervert the estimation practice, and result in meaningless figures.” Inverting the estimation bias on identical delivery outcomes moves the success rate from 5.8% to 94.2% — the metric measures forecasting culture, not project outcomes.

Standish’s own chairman, responding to the authors:

“All data and information in the Chaos reports and all Standish reports should be considered Standish opinion and the reader bears all risk in the use of this opinion.”

To which the authors add that this disclaimer never appeared in the reports themselves.

Market size. Current estimates for the 2026 software testing market include $62.47B, $54.44B and about $59.2B — a spread of $8bn, or 15%, about a year that has already begun. Growth forecasts differ by five percentage points, so the estimates diverge over time. None publishes a reconcilable scope definition, and all sit behind four-figure paywalls.

And one caught in the act. The same report that produces the $2.41T figure contains, in a bulleted list with no citation: “100+ billion new LOC produced worldwide each year → 25 bugs per 1000 LOC injected on average.” An unsourced folk number laundering itself into a document that is subsequently cited as authoritative.

How to check one yourself

Three questions, in order:

1. Can you open the cited document? A surprising share of testing statistics fail here. If the citation is a course note, a private communication, or a gated report with no methodology page, treat the number as unsourced.
2. Does the document contain the number? Several claims in this post survived step one and failed step two — the source exists and says something materially different.
3. Is there a stated sample and scope? “5,971 Microsoft professionals, previous-workday recall” is a statistic. “Industry reports suggest” is not.

Nothing here argues that testing statistics are useless. Google’s flakiness figures, DORA’s benchmarks, CircleCI’s telemetry, Boehm’s actual chart — all real, all useful, all with limits worth knowing. The problem is not the good numbers; it is that they circulate alongside fabrications at identical confidence.

Our own stake, stated plainly: Qualflare sells test observability, and the 100x defect claim is the single most convenient statistic our category could have. It would sit very comfortably in a business case for buying what we sell. It has no source, so we do not use it — and neither should anyone else pitching you a testing tool. The measured version, roughly 5x with wide variance and one large study finding no effect at all, is a weaker argument and a true one. On flakiness specifically, the numbers that do hold up are collected in flaky test statistics and the practical treatment is in the complete guide to flaky tests.

Frequently asked questions

Is it really 100x more expensive to fix a bug in production?

There is no traceable evidence for it. The claim is sourced to 1981 course notes from an “IBM Systems Sciences Institute” that nobody has ever produced, and the real IBM entity of a similar name was an employee training school that published no studies. Barry Boehm’s actual 1976 data shows a median escalation nearer 17x, and Boehm himself revised the figure to about 5:1 for small non-critical systems in 2001. NIST measured roughly 5x, and a 2017 study across 171 projects found no effect at all.

How much time do developers spend on testing and debugging?

Around a fifth to a quarter, by measurement rather than recall. A survey of 5,971 Microsoft professionals put bugfixing at 14% of the workday and testing at 8%. IDE telemetry from 2,443 engineers over two and a half years found developers spend about a quarter of their IDE time on tests — and that they overestimated it by roughly twofold, which is probably where the folklore figures come from.

What percentage of tests are flaky?

Google published two figures that are widely conflated: almost 16% of its tests showed some flakiness at some point, while about 1.5% of all test runs reported a flaky result. Its 2017 analysis broke that down by size — 0.5% of small tests, 1.6% of medium, 14% of large. Note that Google’s own 2020 book states a flaky rate of about 0.15%, ten times lower than the 2016 figure, and the two have never been reconciled.

What are the current DORA benchmarks?

There are none in the current report. DORA’s 2025 report replaced the performance clusters with seven descriptive archetypes carrying no metric thresholds, and the word “elite” does not appear in it. For the familiar four-tier table you have to cite the 2024 report — and be aware that the version circulating online is usually a splice of figures from different years.

How do you check whether a testing statistic is real?

Follow the citation to a document you can open, and check three things: does the source actually contain the number, does it say what the quoter claims, and is there a stated sample. A large share of testing statistics fail at the first step — the cited document either does not exist, is not obtainable, or turns out to be a vendor marketing page citing another vendor marketing page.