Introduction to Visual Regression Testing and getting started with BackstopJS

What is visual regression testing?

Visual Regression Testing (VRT) is a type of software testing that focuses on detecting unintended visual changes in a user interface. Unlike traditional functional testing, which verifies that code behaves correctly, visual regression testing verifies that the appearance of your application remains consistent across changes.

The way it works is straightforward: a visual testing tool takes screenshots of your application's pages or components, then compares them against a set of baseline images (previously approved screenshots). Any differences between the current state and the baseline are flagged as potential regressions.

Consider a scenario where a developer refactors a CSS utility class that appears unrelated to a checkout page. Unknowingly, this change causes the "Place Order" button to shift by a few pixels, making it misaligned on mobile devices. A unit test would not catch this, because the button still functions correctly. A visual regression test, however, would detect the pixel-level difference and alert the team before the change reaches production.

Visual regression testing offers several advantages, but it also comes with trade-offs. Understanding both sides helps you decide whether it fits your team's needs.

Advantages of VRT

• Catching unintended visual changes. The primary benefit is the ability to detect UI bugs that functional tests miss. This includes layout shifts, color changes, font rendering issues, and responsive design breaks.
• Documentation of UI states. Over time, your baseline images become a visual catalog of your application's key states. This can be useful for onboarding new team members or auditing the UI during redesigns.
• Confidence during refactoring. When making large-scale CSS or component changes, visual tests provide a safety net. You can refactor with the assurance that any visual side effects will be caught.
• Cross-browser and cross-device validation. Many visual testing tools, including BackstopJS, allow you to capture screenshots across multiple viewports and browser engines. This helps ensure consistency without manually testing every combination.

Disadvantages of VRT

• False positives from dynamic content. Elements that change between runs (such as timestamps, animations, or third-party ads) can cause tests to fail even when nothing is broken. This requires configuration work to mask or ignore dynamic regions.
• Baseline maintenance overhead. Every intentional UI change requires updating the baseline images. For rapidly evolving projects, this can become tedious. Teams must establish a workflow for reviewing and approving new baselines.
• Not a replacement for other tests. VRT complements unit and integration tests but does not replace them. A page can look correct while functioning incorrectly, and vice versa.
• Storage and CI considerations. Storing hundreds or thousands of baseline images can consume significant disk space. Additionally, running visual tests in CI pipelines may increase build times, especially when testing multiple viewports.

When to adopt Visual Regression Testing

A team should consider adopting VRT when:

• The application has a complex or frequently changing UI, especially if it must be pixel-perfect.
• Manual visual QA is too tedious and becomes a bottleneck before releases.
• The team has experienced UI regressions slipping into production despite having passed QA.
• There is a need to automate testing on multiple browsers, devices, themes, etc.

On the other hand, visual regression testing may be excessive for projects with minimal or no UI and early prototypes with sparse content. Teams may also decide against VRT in CI due to time and cost.

Generally, non-visual testing (especially unit testing) is considered most essential for the average project, while visual testing should be considered as a supplement if required. Alternatively, it's also possible to perform non-automated, manual visual testing only as needed.

Getting started with BackstopJS

BackstopJS is a free and open-source visual regression testing tool for the web. It is relatively easy to set up, can run in Docker, and integrates with popular testing frameworks and CI pipelines. It's by no means the only VRT tool, but we recommend it as an easy starting point.

In the following sections, we'll show you practical examples to help you try VRT using BackstopJS. See the official guide to install it, and get an initial backstop.json configuration file, or try the one provided below.

Sample configuration file

If you use the following file, make sure to edit the URLs in scenarios to match your project.

{
  "id": "my_app",
  "viewports": [
    {
      "label": "phone",
      "width": 375,
      "height": 667
    },
    {
      "label": "tablet",
      "width": 1024,
      "height": 768
    }
  ],
  "onBeforeScript": "playwright/onBefore.js",
  "onReadyScript": "playwright/onReady.js",
  "scenarios": [
    {
      "label": "Homepage",
      "url": "http://localhost:3000/",
      "delay": 0,
      "selectors": [],
      "hideSelectors": [],
      "removeSelectors": [],
      "misMatchThreshold" : 0.1,
      "selectorExpansion": true
    }
  ],
  "paths": {
    "bitmaps_reference": "backstop_data/bitmaps_reference",
    "bitmaps_test": "backstop_data/bitmaps_test",
    "engine_scripts": "backstop_data/engine_scripts",
    "html_report": "backstop_data/html_report",
    "ci_report": "backstop_data/ci_report"
  },
  "report": ["browser", "CI"],
  "engine": "playwright",
  "engineOptions": {
    "args": ["--no-sandbox"]
  },
  "debug": false
}

Key configuration options:

• viewports: defines the screen sizes to test. Add entries for all target devices.
• scenarios: each scenario represents a page or component to test. The url field points to the page, and selectors determines what to capture. Using "document" captures the entire page, while a CSS selector like ".some-component" captures only that element.
• report: specifies output formats. browser outputs human-readable HTML. ci and json print automation-friendly output.
• engine: allows choosing between Playwright and Puppeteer for running scripts during diffs. Puppeteer is used by default, although Playwright is probably preferable at the time of writing thanks to it having richer, more up-to-date features and multiple browser support.

Create a reference image

Before you can detect regressions, you need to create a set of baseline (reference) images. Run:

npx backstop reference

This command launches a headless browser, navigates to each scenario's URL, and saves screenshots to the bitmaps_reference folder. These images represent the "expected" state of your UI.

Run tests

Once reference images exist, you can run visual regression tests:

npx backstop test

BackstopJS captures new screenshots and compares them against the reference images. Any differences beyond a configurable threshold are reported as failures. The HTML report opens in your browser and shows side-by-side comparisons with highlighted differences.

Handle dynamic content

Dynamic content, such as elements generated by JavaScript during runtime, is a common cause of false positives during any form of visual testing. BackstopJS provides several ways to mitigate this:

• Modifying elements: The hideSelectors and removeSelectors fields can be used to hide and remove elements, respectively. Alternatively, custom scripts found inside backstop_data, (e.g. playwright/onReadyScript.js) can be used for arbitrary modifications before capturing.
• Setting a delay: Content that loads asynchronously can be waited for using the delay field.
• Simulating mouse interaction: The hoverSelector and clickSelectors properties can simulate mouse interaction before capturing.

Integrating with CI

BackstopJS can run in CI environments to block or warn about deployments that introduce visual regressions. The ci report format outputs a machine-readable summary:

# Use the report=ci flag if not specified in config:
npx backstop test --report=ci

The exit code will be non-zero if any visual regression is detected, which can fail a CI pipeline. For example, a basic GitHub Actions workflow might look like this:

name: Visual Regression Tests

on: [pull_request]

jobs:
  visual-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout project code
        uses: actions/checkout@v4
      - name: Setup Nodejs
        uses: actions/setup-node@v4
        with:
          node-version: '22'
      - name: Install dependencies
        run: |
          npm ci
          npx playwright install-deps
          npx playwright install
      - run: npx backstop test --report=ci
        if: github.event_name == 'pull_request'

This workflow runs only on pull requests, which helps save time and computing costs.

Although not covered in this article, more sophisticated pipelines may also incorporate new reference image approval, and may save reference images to persistent cloud storage.

Approving changes

When an intentional UI change is made, you need to update the baseline images. BackstopJS provides an approve command:

npx backstop approve

This promotes the current test images to new reference images. It is recommended to review the HTML report before approving to ensure no unintended changes are being accepted.

Closing words

Visual regression testing is a powerful addition to your testing strategy, especially for applications with complex or critical UIs. It catches bugs that functional tests miss and provides confidence during refactors. However, it requires discipline in maintaining baselines and handling dynamic content.

BackstopJS offers a straightforward entry point into visual regression testing. It is easy to configure, supports multiple viewports, and integrates well with CI pipelines. Start by testing a few critical pages, then expand coverage as your team becomes comfortable with the workflow.

Commercial alternatives such as Chromatic and Percy offer features such as parallel testing and review workflows and may be considered over open source options.