Visual regression pipeline

The plan itself is not complicated. A card turns up in three of the sites, so I build one card component that covers the four ways it is actually used, delete the three, and the kit has grown by one. Repeat until there is nothing left to repeat.

The catch is that every one of those deletions edits markup a live site renders, and there is no version of me careful enough to eyeball that across seven of them. Most of these are in production — as much as a side project ever is — so the check had to exist first.

Every page of every site is screenshotted across 12 device and browser profiles (around 530 images a run), and diffed against baselines held in Cloudflare R2.

projects: [
  // Dev profile — pnpm test:visual
  { name: 'desktop-chrome',     use: { ...devices['Desktop Chrome'] } },
  { name: 'iphone-14-pro',      use: { ...devices['iPhone 14 Pro'] } },

  // PR profile — pnpm test:visual:pr
  { name: 'desktop-firefox',    use: { ...devices['Desktop Firefox'] } },
  { name: 'desktop-safari',     use: { ...devices['Desktop Safari'] } },
  { name: 'iphone-14-pro-max',  use: { ...devices['iPhone 14 Pro Max'] } },
  { name: 'iphone-se',          use: { ...devices['iPhone SE'] } },
  { name: 'pixel-7',            use: { ...devices['Pixel 7'] } },
  { name: 'galaxy-s8',          use: { ...devices['Galaxy S8'] } },
  { name: 'ipad-pro-landscape', use: { ...devices['iPad Pro 11 landscape'] } },
  { name: 'ipad-mini',          use: { ...devices['iPad Mini'] } },
]

Runs are pinned to a Docker image. Linux and Windows disagree about rendering at the pixel level — who would have guessed — and a baseline generated on the wrong platform will never match anything again.

services:
  playwright:
    image: snowmaker-playwright:1.58.2   # must equal @playwright/test exactly
    volumes:
      - ./tests/snapshots:/app/tests/snapshots   # baselines, written back to host
      - ./test-results:/app/test-results         # failure PNGs must survive --rm

Most of the work turned out to be making screenshots stable at all. Animation delays had to be zeroed, because prefers-reduced-motion cuts duration and leaves delay alone, so a staggered list sits at opacity zero between the two frames Playwright compares. Lazy images had to be forced to decode rather than merely report themselves complete, which they will do while the pixels are still not painted. Fonts had to be waited on. Until all of that settles, a page diffs against itself, and the failures are the harness rather than the site.

// fullPage expands the viewport, which trips IntersectionObserver and
// starts lazy loads DURING Playwright's two-frame stability check.
for (const img of document.querySelectorAll('img[loading="lazy"]')) {
  img.setAttribute('loading', 'eager');
}

// decode() resolves only once pixel data is ready to paint.
// img.complete resolves earlier, and lies for decoding="async".
await Promise.all([
  ...imgs.map((img) => img.decode().catch(() => {})),
  document.fonts.ready,
]);

So there is a regression net, and all that remains is to use it for a controlled descent into madness while reworking a few hundred components. Turns out that is not much of a developer experience, even with the generated HTML diff reports. I had picked up Electron on a couple of projects by then, so naturally: build a desktop application for a problem I had spent all of ten minutes with.