Product news and testing tips.
Visual tests flake because a screenshot depends on the OS, GPU, font rasterisation, browser build and load timing — none of which your test controls. Playwright's defaults are stricter than most people realise: it tolerates per-pixel colour drift but, with `maxDiffPixels` unset, zero drifting pixels overall. There are only two structural fixes — freeze the rendering environment, or move rendering somewhere that is already frozen.
Key takeaways
- Playwright's `maxDiffPixels` and `maxDiffPixelRatio` are unset by default — the aggregate tolerance is zero.
- Playwright's default headless browser is `chromium-headless-shell`, not the Chromium it uses headed.
- Snapshot filenames encode browser and platform (`-chromium-darwin`) because the format assumes they differ.
- Peer-reviewed 2026 research found visual flakiness is a mean 37.2% of a project's flaky instances — not a false-positive rate.
- Every tool ships a different noise floor, from jest-image-snapshot's 0.01 to Chromatic's 0.063.
A visual regression test compares a screenshot against a stored baseline. The trouble is that a screenshot is not a property of your application — it is a property of your application rendered by a particular browser build, on a particular OS, with particular fonts, on particular hardware. Change any of those and the pixels change, whether or not anything you shipped did.
Playwright’s documentation is unusually blunt about the size of that list:
Browser rendering can vary based on the host OS, version, settings, hardware, power source (battery vs. power adapter), headless mode, and other factors.
Power source. Your laptop being unplugged can change a screenshot.
The defaults are stricter than you think
The most common surprise is not a subtle rendering issue — it is a misread of Playwright’s own defaults.
| Option | Default |
|---|---|
threshold | 0.2 |
maxDiffPixels | unset |
maxDiffPixelRatio | unset |
animations | "disabled" |
scale | "css" |
caret | "hide" |
maskColor | pink #FF00FF |
People read threshold: 0.2 as “20% tolerance” and conclude they have plenty of headroom. It is not that. threshold is a per-pixel colour-difference tolerance in YIQ space, used by the pixelmatch comparator. It decides whether an individual pixel counts as different at all.
How many pixels may differ is governed by maxDiffPixels and maxDiffPixelRatio — and both are unset. So the out-of-the-box contract is: each pixel may drift a little in colour, and zero pixels may actually change. That is a strict test, and it is why suites pass locally and fail the moment they run anywhere else.
A useful comparison of noise floors across tools, because they disagree wildly:
| Tool | Default noise floor |
|---|---|
| Playwright | threshold 0.2, aggregate tolerance 0 |
| jest-image-snapshot | failureThreshold 0, customDiffConfig.threshold 0.01 |
| Chromatic | diffThreshold 0.063, antialiasing ignored by default |
| BackstopJS | misMatchThreshold 0.1 |
| reg-suit | thresholdRate 0, thresholdPixel 0, enableAntialias off |
Note the inversion in the last two rows: Chromatic ignores anti-aliased pixels by default; reg-suit ignores them only if you opt in. Two sensible tools, opposite defaults, and a team migrating between them will see a step change in failure rate that has nothing to do with their UI.
Your headless browser is not your browser
This one is under-appreciated and explains a whole category of “works when I watch it” failures.
Playwright ships two different Chromium binaries. Headed runs use a regular Chromium build; headless runs use a separate chromium headless shell by default. Playwright says so directly, and adds that Chrome and Edge have since moved to a new headless mode closer to headed rendering, which the shell is not — so “expect different behavior in some cases.”
Chrome’s own documentation describes the old split in stronger terms: headless “was a separate, alternate browser implementation that happened to be shipped as part of the same Chrome binary. It didn’t share any of the Chrome browser code.”
So a baseline captured while debugging headed, compared against a headless CI run, is comparing output from two different renderers. That is not flakiness — it is a category error that presents as flakiness.
Fonts flake twice
Fonts break visual tests at two separate stages, and the fixes are different.
At load time. Chromatic documents the race: “Browsers can decide to render HTML in multiple passes when custom fonts are used… This behavior can cause a test to render without the custom font or use different fonts across repeated runs, making the test unstable.” The fix is to wait for fonts to be ready before capturing — not a fixed sleep.
At rasterisation. Even with the right font loaded, the glyphs are drawn differently on different systems. Chromium’s own web-test infrastructure goes to remarkable lengths here: “We try to match the Windows render tree output exactly by matching font metrics and widget metrics… Your main display’s ‘Color Profile’ is also changed to make sure color correction by ColorSync matches what is expected in the pixel tests.”
When the Chromium project has to override your display’s colour profile to make its own pixel tests reproducible, a test suite that assumes two laptops will rasterise identically is optimistic.
This is also why antialiasing settings matter so much: Percy’s Strict sensitivity is defined as highlighting “every pixel difference, including image artifacts and font smoothing/antialiasing” — which is exactly the class of difference that varies by machine.
The statistic everyone quotes, and the one you should
A “10–40% false-positive rate for visual tests” circulates widely. It has no primary source. The phrasing traces to a vendor blog post — from a company selling a product claiming zero false positives — with no study, dataset, methodology or citation behind it, hand-waving at “sources and configurations.” It should not be repeated, including by us.
There is real research, and it measures something adjacent but distinct. Pei, Sohn and Papadakis, An Empirical Study of Web Visual Flakiness, Journal of Systems and Software, 2026, analysed 262 cases — 144 from 31 open-source web projects and 118 from Chromium. Findings worth citing:
- Causes split 59.9% structure-related and 40.1% style-related.
- Flakiness frequency was 3.7‰ for visual commits against 0.8‰ for non-visual ones — visual changes are roughly four to five times likelier to introduce flakiness.
- Visual flakiness accounted for a mean 37.2% (median 33.3%) of a project’s flaky instances.
Be precise about that last number, because it is easy to misuse. “37.2% of a project’s flaky instances are visual” is not “37.2% of visual test failures are false positives” — different denominator, different claim. It supports the argument that visual flake is a disproportionate share of all flake, which is the more useful point anyway.
The two structural fixes
Threshold tuning is symptom management. There are only two approaches that address the cause, and they are opposites.
Freeze the environment. Render in a container so every run uses the same OS, fonts and browser build. BackstopJS is explicit about this, listing “Integrated Docker rendering — to eliminate cross-platform rendering shenanigans” as a feature, with a --docker flag. Vitest recommends the same: visual tests “are most reliable when run in a standardized and tightly controlled environment.”
One correction worth making, since the advice is often misattributed: Playwright does not recommend Docker for this. Its Docker page describes an image for testing and development and never mentions screenshots, fonts or rendering consistency. Playwright’s actual answer is per-platform baselines — which is why snapshot filenames look like example-test-1-chromium-darwin.png. The format encodes browser and platform because the docs assume they differ: “Screenshots differ between browsers and platforms due to different rendering, fonts and more, so you will need different snapshots for them.”
The filename is telling you the problem before you hit it.
Move the rendering. Percy takes the other route: it does not diff your test browser’s screenshots at all. Its SDKs serialise the DOM and Percy re-renders it server-side, in one controlled environment, with JavaScript disabled by default and CSS animations frozen. Whatever your CI machine’s fonts and GPU are doing becomes irrelevant, because it is not the thing being photographed.
Both approaches accept the same premise: you cannot make heterogeneous machines render identically, so stop trying and render in one place.
A note on “Visual AI”
Applitools markets Visual AI as detecting “only meaningful visual differences that could impact the user experience” using “advanced image comparison algorithms that have been developed and refined over more than a decade” which “simulate human vision.”
Two things worth stating accurately. Its documented match levels are Strict, Layout, Ignore colors, Dynamic, Exact and None — and the default for web tests is Dynamic, not Strict, which is the opposite of what most write-ups say. And while “AI” is in the name, the documentation describes image comparison algorithms simulating human vision; it does not claim a neural network or trained model, and no published false-positive figure exists. Treat the capability claim as a vendor claim, which is what it is.
Where this sits
Visual flakiness is flakiness with an unusually legible cause. A normal flaky test leaves you guessing between timing, ordering and shared state; a visual one usually resolves to environment, timing or tolerance, and the JSS data suggests it is a disproportionate share of the total.
What it shares with every other kind is the diagnostic requirement: one failed run cannot tell you whether a diff is a real regression or your CI image picking up a font update. That needs the same test’s history — how often it diffs, whether it started at a specific commit, whether it correlates with a runner — which is the general problem covered in the complete guide to flaky tests and measured as a flake rate.
Our own stake, stated plainly: Qualflare ingests results from CI and scores flakiness from history, so a visual test that diffs intermittently shows up in the same place as any other unreliable test. It does not render screenshots, does not compare images, and does not replace Playwright, Percy or Chromatic — those do the visual work and this reads what they emit. If your problem is that a specific test diffs on every CI run, the fix is in this post, not in a dashboard.
Frequently asked questions
Why do my Playwright screenshot tests fail in CI but pass locally?
Because the screenshot depends on things your test does not control — operating system, GPU and drivers, font rasterisation, browser build, screen scaling and colour profile. Playwright’s documentation lists host OS, version, settings, hardware, power source, and headless mode among the variables. Its own answer is to run tests in the same environment that generated the baselines, which is why snapshot filenames include the platform.
What is Playwright’s default screenshot tolerance?
Stricter than most people expect. threshold defaults to 0.2, which is a per-pixel colour-difference tolerance in YIQ space. But maxDiffPixels and maxDiffPixelRatio are both unset by default, so the number of pixels allowed to differ is effectively zero. Playwright tolerates a slightly different shade on a pixel, but not a pixel that changed.
How do you stop antialiasing from breaking visual tests?
Different tools take opposite defaults, so check yours. Chromatic detects anti-aliased pixels and ignores them by default; reg-suit has enableAntialias off by default, meaning it does not ignore them unless you opt in. Percy’s Strict sensitivity explicitly highlights font smoothing and antialiasing. The durable fix is not a threshold but rendering in a single fixed environment.
Is there a real statistic for visual test false-positive rates?
No published primary source measures one. A widely-circulated 10–40% range traces back to a vendor blog post with no methodology, dataset or citation. What does exist is peer-reviewed work in the Journal of Systems and Software (2026) analysing 262 cases, which found visual flakiness accounts for a mean 37.2% of a project’s flaky instances — a different measurement with a different denominator.
Does running visual tests in Docker fix flakiness?
It fixes the cross-machine half, which is usually the larger half. BackstopJS recommends it explicitly, listing integrated Docker rendering to eliminate cross-platform rendering differences, and Vitest recommends a standardised controlled environment. Note that Playwright’s own Docker documentation does not mention screenshots or rendering consistency — its stated answer is per-platform baselines instead.
