Visual regression

Catch appearance changes assertions miss: baselines, the five diff statuses, the approve-versus-reject decision, and how to stop diffs being flaky.

A test can pass every assertion while the page looks broken — a stylesheet that failed to load, a layout that collapsed at one width, a button that is now white on white. Assertions check structure; visual regression checks appearance.

How it works

Add a screenshot step where you want the page compared. On the first run there is nothing to compare against, so the capture is recorded as new. Approve it and it becomes the baseline for that name.

From then on, every run captures the same shot and compares it pixel-by-pixel against the baseline. A difference produces a diff — the baseline, the new capture, and an image highlighting exactly which pixels changed.

Comparison is pixel-based (pixelmatch/pngjs), not a perceptual model. It sees everything, including the things you do not care about — which is why the housekeeping below matters more than the feature itself.

Statuses

Every comparison lands in one of five states:

Status	Meaning
`new`	No baseline for this name yet. Nothing to compare.
`match`	Identical to the baseline within tolerance.
`mismatch`	Differs from the baseline. Needs a human decision.
`approved`	You reviewed a mismatch and accepted it. The new image becomes the baseline.
`rejected`	You reviewed a mismatch and called it a bug. The baseline is unchanged.

approved and rejected are the two halves of the review, and the distinction is the whole point of the feature. Approve means "the design changed, this is the new correct appearance" — the baseline moves. Reject means "the design did not change, this is a regression" — the baseline stays, so the next run flags it again until it is fixed.

Approving a diff you have not looked at converts the feature into a system that records whatever happened. It is worth being disciplined here in a way that is not true of most test maintenance.

Reviewing a diff

The diff viewer shows baseline, actual and difference together, with the changed regions marked. Small text and antialiasing differences are usually a rendering artifact; a shifted block or a colour change usually is not.

Naming screenshots

The name field identifies the baseline, so it has to be stable across runs — that is what makes run 40's capture comparable to run 1's. Use names that describe the state (checkout-empty, checkout-with-items), not the run.

Names are sanitized before they touch the filesystem, so a name with a slash or a .. in it cannot escape the artifact directory.

What makes visual tests flaky, and what to do

Pixel comparison is exact, so anything genuinely variable will differ every run:

Timestamps, relative dates, IDs. Pin them with set-clock, or seed fixed data, or frame the screenshot to exclude them.
Animations and transitions. Screenshot after they settle — a wait for the end state, or the "Wait for Loading" preset.
Third-party embeds and ads. Block them with mock-route.
Font loading. A shot taken mid-swap differs from one taken after.
Viewport differences. A baseline is only comparable to a capture at the same viewport. Changing a test's viewport invalidates its baselines.

A visual test that mismatches on every run teaches the team to approve without looking, which is worse than not having it. Spend the effort on making the shot deterministic, or narrow what it covers.

Where to use it

Visual regression pays off on pages where appearance is the product — a marketing page, a dashboard's layout, a themed component library, a print/PDF view. It pays off least on pages dense with live data, where you will spend more time managing diffs than reading them.