Visual Regression Testing on CI: Mastering Playwright Snapshot Tests for Consistent UI

In our previous blog, “Diving Deeper into Visual Testing,” we explored advanced snapshot testing techniques to enhance the stability and reliability of WorkWave, TechWave Solutions’ flagship project. Now, let’s delve into advanced configuration and CI/CD integration to further streamline and automate our visual regression testing processes, with a focus on maintaining consistency in snapshot tests across different environments.

Revisiting TechWave Solutions

With a solid foundation in advanced visual testing, Sarah and her team at TechWave Solutions were well-equipped to catch visual bugs before they reached production. However, to keep up with the increasing complexity of WorkWave, they needed to automate and integrate these tests into their CI/CD pipeline. This would ensure consistent testing and quicker feedback, helping to maintain UI consistency across various environments.

Advanced Configuration Techniques

1. Custom Configuration for Different Environments

Liam, the senior developer, emphasised the importance of tailoring Playwright configurations for different environments, such as development, staging, and production.

Example: Custom Configuration File

// playwright.config.js
module.exports = {
......
  projects: [
    {
      name: 'dev',
      use: {
        baseURL: 'https://dev.workwave.com',
        headless: false,
      },
    },
    {
      name: 'staging',
      use: {
        baseURL: 'https://staging.workwave.com',
        headless: true,
      },
    },
    {
      name: 'prod',
      use: {
        baseURL: 'https://www.workwave.com',
        headless: true,
      },
    },
  ],
};

2. Advanced Reporter Configuration

Sarah highlighted the importance of detailed reporting for better insights into test results. Playwright supports various reporters, including JSON and HTML.

Example: Configuring Multiple Reporters

// playwright.config.js
module.exports = {
......
  reporter: [
    ['list'],
    ['json', { outputFile: 'test-results/test-results.json' }],
    ['html', { outputFolder: 'test-results/report', open: 'never' }],
    ['junit', { outputFile: 'test-results/results.xml' }],
  ],
};

3. Storing baseline Snapshots

Storing baseline snapshots ensures consistency in visual testing. These snapshots should be accessible by both local and CI environments. Hence we will set a common Snapshot Dir as below:

// playwright.config.js
module.exports = {
......
  snapshotDir: '../../../snapshots'
};

4. Ensuring Consistency Between Local & CI Environments with Docker

To maintain consistency between local and CI environments, it’s essential to ensure that both have the same dependencies and settings. Using Docker can help create a consistent testing environment.

Dockerfile for Playwright

Example: Dockerfile for Playwright

# Dockerfile
FROM mcr.microsoft.com/playwright:1.43.1-focal

WORKDIR /src

COPY package.json package-lock.json ./
RUN npm install

COPY . .

RUN npx playwright install

Docker Compose for Visual Tests

Example: docker-compose.visual.yml

# docker-compose.visual.yml
version: '3.8'

services:
  visual-test:
    build:
      context: .
      dockerfile: Dockerfile
    volumes:
      - .:/src

Running Tests Locally with Docker

To run tests locally, use the following commands:

sudo docker-compose -f docker-compose.visual.yml build

Once image is built, Run the Container and Execute Tests

docker-compose -f docker-compose.visual.yml run --rm visual-test npx playwright test --project="staging"

Running Tests on CI with GitHub Actions

Note: Before we run Visual Tests on CI, make sure all the tests has generated baseline snapshots in the /snapshots directory.

To run the tests on CI, we would need to use some CI CD tool. We will use github actions in this example as recommended by Playwright here.

Example: GitHub Actions Workflow with same playwright image version

# .github/workflows/playwright-docker.yml
name: Playwright Tests with Docker

on:
  push:
    branches:
      - main
      - 'release/*'
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:1.43.1-focal

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
      - uses: actions/setup-node@v4
        with:
          node-version: 18
      - name: Install dependencies
        run: npm ci
      - name: Run Playwright tests
        run: npx playwright test --project="staging"
        env:
         HOME: /root

So, once you push your commit in the main or release/** branches, above workflow will be triggered & it will start executing testcases.

Advanced Testing Strategies

Parallel Testing

Sarah explained the importance of parallel testing to speed up the feedback loop. Playwright supports running tests in parallel by default, but it’s essential to ensure tests are independent and isolated.

Example: Enabling Parallel Testing

// playwright.config.js
module.exports = {
  workers: 4,  // Adjust based on your CI server capacity
  // Other configurations...
};

Handling Flaky Tests

To handle flaky tests, Liam showed how to configure retries and include detailed logging to diagnose issues.

Example: Configuring Retries and Logging

// playwright.config.js
module.exports = {
  retries: 2,
  reporter: [
    ['dot'],
    ['json', { outputFile: 'test-results.json' }],
    ['html', { open: 'never' }],
    ['junit', { outputFile: 'test-results/results.xml' }],
  ],
  use: {
    trace: 'on-first-retry',  // Enable tracing for flaky tests
  },
};

Conclusion: Automating and Optimizing Testing Workflows

Integrating advanced configuration and CI/CD workflows significantly enhanced TechWave Solutions’ ability to maintain a reliable and consistent UI for WorkWave. By automating their testing process, they reduced manual efforts and ensured quicker feedback, ultimately leading to a more stable and high-quality product.

As they continue to refine their testing strategies, the value of automated visual testing becomes increasingly evident. For those keen on diving deeper, further exploration of advanced CI/CD configurations and test optimisation techniques is highly recommended.

Happy testing!