Automated Visible Regression Testing With Playwright

Evaluating visible artifacts generally is a highly effective, if fickle, strategy to automated testing. Playwright makes this appear easy for web sites, however the particulars would possibly take a bit of finessing.

Current downtime prompted me to scratch an itch that had been plaguing me for some time: The model sheet of a web site I preserve has grown just a bit unwieldy as we’ve been including code whereas exploring new options. Now that we’ve a greater concept of the necessities, it’s time for inside CSS refactoring to pay down a few of our technical debt, making the most of fashionable CSS options (like utilizing CSS nesting for extra apparent construction). Extra importantly, a cleaner basis ought to make it simpler to introduce that darkish mode function we’re sorely missing so we are able to lastly respect customers’ most well-liked coloration scheme.

Nevertheless, being of the apprehensive persuasion, I used to be reluctant to make giant modifications for worry of unwittingly introducing bugs. I wanted one thing to protect towards visible regressions whereas refactoring — besides which means snapshot testing, which is notoriously gradual and brittle.

On this context, snapshot testing means taking screenshots to ascertain a dependable baseline towards which we are able to evaluate future outcomes. As we’ll see, these artifacts are influenced by a large number of things that may not at all times be totally controllable (e.g. timing, variable {hardware} assets, or randomized content material). We even have to keep up state between take a look at runs, i.e. save these screenshots, which complicates the setup and means our take a look at code alone doesn’t totally describe expectations.

Having procrastinated with out a extra agreeable answer revealing itself, I lastly got down to create what I assumed could be a fast spike. In spite of everything, this wouldn’t be a part of the common take a look at suite; only a one-off utility for this specific refactoring activity.

Happily, I had obscure recollections of previous analysis and rapidly rediscovered Playwright’s built-in visible comparability function. As a result of I attempt to choose dependencies fastidiously, I used to be glad to see that Playwright appears to not depend on many exterior packages.

Setup

The really useful setup with npm init playwright@newest does a good job, however my minimalist style had me set every little thing up from scratch as an alternative. This do-it-yourself strategy additionally helped me perceive how the totally different items match collectively.

On condition that I anticipate snapshot testing to solely be used on uncommon events, I wished to isolate every little thing in a devoted subdirectory, referred to as take a look at/visible; that will probably be our working listing from right here on out. We’ll begin with package deal.json to declare our dependencies, including just a few helper scripts (spoiler!) whereas we’re at it:

{
  "scripts":  true"
  ,
  "devDependencies": {
    "@playwright/take a look at": "^1.49.1"
  }
}

For those who don’t need node_modules hidden in some subdirectory but additionally don’t need to burden the foundation mission with this rarely-used dependency, you would possibly resort to manually invoking npm set up --no-save @playwright/take a look at within the root listing when wanted.

With that in place, npm set up downloads Playwright. Afterwards, npx playwright set up downloads a variety of headless browsers. (We’ll use npm right here, however you would possibly choose a distinct package deal supervisor and activity runner.)

We outline our take a look at atmosphere by way of playwright.config.js with a few dozen primary Playwright settings:

import { defineConfig, gadgets } from "@playwright/take a look at";

let BROWSERS = ["Desktop Firefox", "Desktop Chrome", "Desktop Safari"];
let BASE_URL = "http://localhost:8000";
let SERVER = "cd ../../dist && python3 -m http.server";

let IS_CI = !!course of.env.CI;

export default defineConfig({
  testDir: "./",
  fullyParallel: true,
  forbidOnly: IS_CI,
  retries: 2,
  employees: IS_CI ? 1 : undefined,
  reporter: "html",
  webServer: {
    command: SERVER,
    url: BASE_URL,
    reuseExistingServer: !IS_CI
  },
  use: {
    baseURL: BASE_URL,
    hint: "on-first-retry"
  },
  initiatives: BROWSERS.map(ua => ({
    identify: ua.toLowerCase().replaceAll(" ", "-"),
    use: { ...gadgets[ua] }
  }))
});

Right here we anticipate our static web site to already reside throughout the root listing’s dist folder and to be served at localhost:8000 (see SERVER; I choose Python there as a result of it’s broadly accessible). I’ve included a number of browsers for illustration functions. Nonetheless, we’d scale back that quantity to hurry issues up (thus our easy BROWSERS record, which we then map to Playwright’s extra elaborate initiatives knowledge construction). Equally, steady integration is YAGNI for my specific situation, in order that complete IS_CI dance may very well be discarded.

Seize and evaluate

Let’s flip to the precise exams, beginning with a minimal pattern.take a look at.js file:

import { take a look at, anticipate } from "@playwright/take a look at";

take a look at("house web page", async ({ web page }) => {
  await web page.goto("/");
  await anticipate(web page).toHaveScreenshot();
});

npm take a look at executes this little take a look at suite (based mostly on file-name conventions). The preliminary run at all times fails as a result of it first must create baseline snapshots towards which subsequent runs evaluate their outcomes. Invoking npm take a look at as soon as extra ought to report a passing take a look at.

Altering our web site, e.g. by recklessly messing with construct artifacts in dist, ought to make the take a look at fail once more. Such failures will supply numerous choices to check anticipated and precise visuals:

We are able to additionally examine these baseline snapshots straight: Playwright creates a folder for screenshots named after the take a look at file (pattern.take a look at.js-snapshots on this case), with file names derived from the respective take a look at’s title (e.g. home-page-desktop-firefox.png).

Producing exams

Getting again to our authentic motivation, what we would like is a take a look at for each web page. As a substitute of arduously writing and sustaining repetitive exams, we’ll create a easy internet crawler for our web site and have exams generated robotically; one for every URL we’ve recognized.

Playwright’s world setup permits us to carry out preparatory work earlier than take a look at discovery begins: Decide these URLs and write them to a file. Afterward, we are able to dynamically generate our exams at runtime.

Whereas there are different methods to move knowledge between the setup and test-discovery phases, having a file on disk makes it straightforward to change the record of URLs earlier than take a look at runs (e.g. briefly ignoring irrelevant pages).

Web site map

Step one is to increase playwright.config.js by inserting globalSetup and exporting two of our configuration values:

export let BROWSERS = ["Desktop Firefox", "Desktop Chrome", "Desktop Safari"];
export let BASE_URL = "http://localhost:8000";

// and so forth.

export default defineConfig({
  // and so forth.
  globalSetup: require.resolve("./setup.js")
});

Though we’re utilizing ES modules right here, we are able to nonetheless depend on CommonJS-specific APIs like require.resolve and __dirname. It seems there’s some Babel transpilation occurring within the background, so what’s really being executed might be CommonJS? Such nuances generally confuse me as a result of it isn’t at all times apparent what’s being executed the place.

We are able to now reuse these exported values inside a newly created setup.js, which spins up a headless browser to crawl our web site (simply because that’s simpler right here than utilizing a separate HTML parser):

import { BASE_URL, BROWSERS } from "./playwright.config.js";
import { createSiteMap, readSiteMap } from "./sitemap.js";
import playwright from "@playwright/take a look at";

export default async perform globalSetup(config) {
  // solely create web site map if it would not exist already
  attempt {
    readSiteMap();
    return;
  } catch(err) {}

  // launch browser and provoke crawler
  let browser = playwright.gadgets[BROWSERS[0]].defaultBrowserType;
  browser = await playwright[browser].launch();
  let web page = await browser.newPage();
  await createSiteMap(BASE_URL, web page);
  await browser.shut();
}

That is pretty boring glue code; the precise crawling is going on inside sitemap.js:

• createSiteMap determines URLs and writes them to disk.
• readSiteMap merely reads any beforehand created web site map from disk. This will probably be our basis for dynamically producing exams. (We’ll see later why this must be synchronous.)

Happily, the web site in query supplies a complete index of all pages, so my crawler solely wants to gather distinctive native URLs from that index web page:

perform extractLocalLinks(baseURL) {
  let urls = new Set();
  let offset = baseURL.size;
  for(let { href } of doc.hyperlinks) {
    if(href.startsWith(baseURL)) {
      let path = href.slice(offset);
      urls.add(path);
    }
  }
  return Array.from(urls);
}

Wrapping that in a extra boring glue code offers us our sitemap.js:

import { readFileSync, writeFileSync } from "node:fs";
import { be part of } from "node:path";

let ENTRY_POINT = "/matters";
let SITEMAP = be part of(__dirname, "./sitemap.json");

export async perform createSiteMap(baseURL, web page) {
  await web page.goto(baseURL + ENTRY_POINT);
  let urls = await web page.consider(extractLocalLinks, baseURL);
  let knowledge = JSON.stringify(urls, null, 4);
  writeFileSync(SITEMAP, knowledge, { encoding: "utf-8" });
}

export perform readSiteMap() {
  attempt {
    var knowledge = readFileSync(SITEMAP, { encoding: "utf-8" });
  } catch(err) {
    if(err.code === "ENOENT") {
      throw new Error("lacking web site map");
    }
    throw err;
  }
  return JSON.parse(knowledge);
}

perform extractLocalLinks(baseURL) {
  // and so forth.
}

The fascinating bit right here is that extractLocalLinks is evaluated throughout the browser context — thus we are able to depend on DOM APIs, notably doc.hyperlinks — whereas the remaining is executed throughout the Playwright atmosphere (i.e. Node).

Exams

Now that we’ve our record of URLs, we principally simply want a take a look at file with a easy loop to dynamically generate corresponding exams:

for(let url of readSiteMap()) {
  take a look at(`web page at ${url}`, async ({ web page }) => {
    await web page.goto(url);
    await anticipate(web page).toHaveScreenshot();
  });
}

That is why readSiteMap needed to be synchronous above: Playwright doesn’t presently assist top-level await inside take a look at recordsdata.

In apply, we’ll need higher error reporting for when the positioning map doesn’t exist but. Let’s name our precise take a look at file viz.take a look at.js:

import { readSiteMap } from "./sitemap.js";
import { take a look at, anticipate } from "@playwright/take a look at";

let sitemap = [];
attempt {
  sitemap = readSiteMap();
} catch(err) {
  take a look at("web site map", ({ web page }) => {
    throw new Error("lacking web site map");
  });
}

for(let url of sitemap) {
  take a look at(`web page at ${url}`, async ({ web page }) => {
    await web page.goto(url);
    await anticipate(web page).toHaveScreenshot();
  });
}

Getting right here was a little bit of a journey, however we’re just about accomplished… except we’ve to cope with actuality, which generally takes a bit extra tweaking.

Exceptions

As a result of visible testing is inherently flaky, we generally have to compensate by way of particular casing. Playwright lets us inject customized CSS, which is commonly the simplest and only strategy. Tweaking viz.take a look at.js…

// and so forth.
import { be part of } from "node:path";

let OPTIONS = {
  stylePath: be part of(__dirname, "./viz.tweaks.css")
};

// and so forth.
  await anticipate(web page).toHaveScreenshot(OPTIONS);
// and so forth.

… permits us to outline exceptions in viz.tweaks.css:

/* suppress state */
major a:visited {
  coloration: var(--color-link);
}

/* suppress randomness */
iframe[src$="/articles/signals-reactivity/demo.html"] {
  visibility: hidden;
}

/* suppress flakiness */
physique:has(h1 a[href="/wip/unicode-symbols/"]) {
  major tbody > tr:last-child > td:first-child {
    font-size: 0;
    visibility: hidden;
  }
}

:has() strikes once more!

Web page vs. viewport

At this level, every little thing appeared hunky-dory to me, till I noticed that my exams didn’t really fail after I had modified some styling. That’s not good! What I hadn’t taken into consideration is that .toHaveScreenshot solely captures the viewport somewhat than the whole web page. We are able to rectify that by additional extending playwright.config.js.

export let WIDTH = 800;
export let HEIGHT = WIDTH;

// and so forth.

  initiatives: BROWSERS.map(ua => ({
    identify: ua.toLowerCase().replaceAll(" ", "-"),
    use: {
      ...gadgets[ua],
      viewport: {
        width: WIDTH,
        top: HEIGHT
      }
    }
  }))

…after which by adjusting viz.take a look at.js‘s test-generating loop:

import { WIDTH, HEIGHT } from "./playwright.config.js";

// and so forth.

for(let url of sitemap) {
  take a look at(`web page at ${url}`, async ({ web page }) => {
    checkSnapshot(url, web page);
  });
}

async perform checkSnapshot(url, web page) {
  // decide web page top with default viewport
  await web page.setViewportSize({
    width: WIDTH,
    top: HEIGHT
  });
  await web page.goto(url);
  await web page.waitForLoadState("networkidle");
  let top = await web page.consider(getFullHeight);

  // resize viewport for earlier than snapshotting
  await web page.setViewportSize({
    width: WIDTH,
    top: Math.ceil(top)
  });
  await web page.waitForLoadState("networkidle");
  await anticipate(web page).toHaveScreenshot(OPTIONS);
}

perform getFullHeight() {
  return doc.documentElement.getBoundingClientRect().top;
}

Observe that we’ve additionally launched a ready situation, holding till there’s no community site visitors for some time in a crude try to account for stuff like lazy-loading photos.

Bear in mind that capturing the whole web page is extra resource-intensive and doesn’t at all times work reliably: You might need to cope with structure shifts or run into timeouts for lengthy or asset-heavy pages. In different phrases: This dangers exacerbating flakiness.

Conclusion

A lot for that fast spike. Whereas it took extra effort than anticipated (I imagine that’s referred to as “software program growth”), this would possibly really remedy my authentic drawback now (not a standard function of software program as of late). After all, shaving this yak nonetheless leaves me itchy, as I’ve but to do the precise work of scratching CSS with out breaking something. Then comes the true problem: Retrofitting darkish mode to an current web site. I simply would possibly want extra downtime.

---

Automated Visible Regression Testing With Playwright initially printed on CSS-Tips, which is a part of the DigitalOcean household. It’s best to get the publication.