Visual testing

Visual testing catches what a functional test passes straight through: a button that moved, a font that fell back, a banner covering the page. Your Playwright tests capture screenshots, sekee-cli sends them to Sekee, and Sekee compares each one against an approved baseline. Someone looks at what changed on the Visual Testing page and approves it or rejects it.

The baselines live in Sekee, not in your repository. There are no image files to commit, to keep up to date in a pull request, or to merge.

:::info Plan Visual testing is a Business plan feature. On other plans the sidebar has no Visual Testing entry, the page says visual testing is not part of your plan, and the upload is refused with 403, so sekee-cli snapshots exits 1. :::

The shape of it

Playwright test       snapshot(page, 'checkout')     writes a PNG to test-results/
sekee-cli snapshots   sends the PNGs to the run      Sekee compares each to its baseline
Visual Testing        a member approves or rejects   approving makes the capture the baseline

A snapshot is one screenshot in one run. Sekee knows it by four things:

• its name, the one you give it in the test;
• the Playwright project it ran in, such as chromium or Mobile Safari;
• the viewport, such as 1280x720;
• the platform it was captured on: linux, darwin (macOS) or win32.

Change any of them and it is a different snapshot, with a baseline of its own. Baselines also belong to a Sekee project, so two projects never share one.

The platform is in there because operating systems render fonts differently. A capture from a Mac and a capture from Linux CI are different snapshots, so a run from somebody's laptop never marks every Linux baseline as changed. It also means you should capture in one fixed environment (see Render in a fixed environment).

Setting it up

Install the helper

The helper ships in the same package as the CLI. Install it in your test project:

npm install --save-dev @sekee/sekee-cli

Capture snapshots in your tests

Call snapshot(page, name) wherever you want a picture:

import { test } from '@playwright/test'
import { snapshot } from '@sekee/sekee-cli/playwright'

test('inventory shows items', async ({ page }) => {
  await page.goto('/inventory')
  await snapshot(page, 'inventory')
})

The page is passed in rather than taken from a fixture, so serial suites that share one page from beforeAll work the same way. The helper finds the running test on its own:

import { test, Page } from '@playwright/test'
import { snapshot } from '@sekee/sekee-cli/playwright'

test.describe('Checkout', () => {
  test.describe.configure({ mode: 'serial' })
  let page: Page

  test.beforeAll(async ({ browser }) => {
    page = await browser.newPage()
    await page.goto('/cart')
  })

  test('shows the cart', async () => {
    await snapshot(page, 'checkout-cart')
  })

  test('asks for payment', async () => {
    await page.getByRole('button', { name: 'Checkout' }).click()
    await snapshot(page, 'checkout-payment', { mask: [page.getByTestId('order-number')] })
  })
})

Nothing is compared during the test, so a snapshot never fails one. Each call writes <project>__<name>.png, and a .json file describing it, to test-results/sekee-snapshots. Playwright empties test-results at the start of every run, so images from an earlier run are never sent with this one.

• The name is required, at most 200 characters, and must be unique within a Playwright project. Two tests using one name overwrite each other's image, and the helper prints a warning when they do.
• To write somewhere else, set SEKEE_SNAPSHOTS_DIR or the dir option. Playwright does not empty that directory, so clear it before each run, and pass the same directory to sekee-cli snapshots --dir.

Every capture is a full-page screenshot with animations stopped, the text caret hidden, and CSS pixels rather than device pixels. The third argument takes:

Option	Default	Description
fullPage	true	Capture the whole page rather than the viewport
mask		Locators to paint over: clocks, avatars, anything that changes per run
maskColor	#FF00FF	Colour of the masked areas
style		CSS applied while capturing, for example to hide a cookie banner
clip		Capture only { x, y, width, height }
timeout		Screenshot timeout in milliseconds
threshold	0.1	Per-pixel colour tolerance, from 0 to 1 (see Tolerances)
maxDiffPixels	0	How many differing pixels still count as unchanged
maxDiffPixelRatio	0	Share of differing pixels, from 0 to 1, that still counts as unchanged
dir		Directory to write to

Render in a fixed environment

Fonts, anti-aliasing and scrollbars render differently on macOS, Windows and Linux, and between Linux machines with different fonts installed. Capture in one place: run the suite in the official Playwright Docker image, pinned to the version of @playwright/test you use.

jobs:
  test:
    runs-on: ubuntu-latest
    container: mcr.microsoft.com/playwright:v1.58.2-noble

To capture on your own machine, run the same image:

docker run --rm --ipc=host -v "$PWD":/work -w /work mcr.microsoft.com/playwright:v1.58.2-noble npx playwright test

Send them from CI

Snapshots go to the same automation run as your reports. The order is init, the tests, thread as usual, then snapshots, then complete:

# 1. Open a run
sekee-cli init --url https://api.sekee.io --name "PR #42" --projectID 1 --source "CI"
# → Run ID: 5

# 2. Run the tests; the helper writes the snapshots
npx playwright test

# 3. Send the report, as you do today
sekee-cli thread --url https://api.sekee.io --runId 5 --name "Playwright" --report "./results.xml"

# 4. Send the snapshots for comparison
sekee-cli snapshots --url https://api.sekee.io --runId 5 --fail-on-changes

# 5. Close the run
sekee-cli complete --url https://api.sekee.io --runId 5

snapshots prints what Sekee found and a link to review it:

✔ Found 40 snapshot(s) in test-results/sekee-snapshots.
Branch: feature/checkout, default branch: main
✔ 40 snapshot(s): 36 unchanged, 2 changed, 1 new, 1 failed

Visual: 40 snapshot(s) — 36 unchanged, 2 changed, 1 new, 1 failed
⚠ checkout-payment (chromium) could not be compared: …
Review: https://app.sekee.io/acme/visual/5

With parallel shards, each shard sends its own snapshots after its report. Run both steps even when the tests failed (if: always() in GitHub Actions), so a red shard's snapshots are still compared. CI recipes has complete pipelines, and the CLI reference every flag.

If you already use upload-reports, it creates a completed run and prints its ID; pass that to snapshots --runId. A completed run takes snapshots until it is 24 hours old.

The API token your pipeline already uses is enough. The upload's answer carries the results, so CI never has to read anything back.

Failing the build on a change

--fail-on-changes makes the step exit 1 when a snapshot failed, or changed and nobody has approved it. A rejected change still fails. A new snapshot never fails it: a first capture is a proposal for a baseline, approved in Sekee like a change. Without the flag, snapshots exits 1 only when some snapshots could not be sent at all.

The loop then goes:

1. CI goes red at the snapshots step and prints the review link.
2. Someone opens it, approves what was meant to change, and rejects what was not.
3. You re-run the job. Once every change in it is approved, the snapshots step passes.

Either way of re-running works:

• Re-running the failed job (in GitHub Actions, Re-run failed jobs) sends the snapshots to the same run. Sekee answers with each snapshot's result and what was decided since, and an approved change no longer fails the step.
• Re-running everything (Re-run all jobs) opens a new run, which compares against the baselines just approved and finds them unchanged.

A run's verdict for a snapshot is its first capture. A re-run that captures a different image under the same name is answered with the run's first capture, and a warning; the new image is not compared.

A rejected change stays red on purpose. Re-running that run answers with the rejection again, and later runs compare against the same baseline, so the job goes green on the first run after the page is fixed.

Reviewing changes

Finding a run

Visual Testing is in the sidebar's Automation group, below Error Tracking. It lists the runs of the project you have open that captured snapshots, newest first, with a status for each. Filter by branch, or choose Needs review to see only the runs with something nobody has decided on yet.

Nothing about snapshots appears on the Automation pages. A run's report stays there; its snapshots are here.

Reading a run

Open a run to review it. The tabs divide its snapshots, and the biggest change comes first within each:

Tab	Shows
Needs review	Changed and new snapshots nobody has approved or rejected yet
Changed	Every snapshot that differed from its baseline, decided or not
New	Every snapshot that had no baseline
Approved	The ones somebody approved
Rejected	The ones somebody rejected
Unchanged	The ones that matched their baseline
Failed	The ones that could not be compared, with the reason
All	Everything in the run

A snapshot with a baseline can be compared three ways:

Mode	Shows
Side by side	The baseline and this run's capture next to each other
Slider	The baseline under the capture. Drag the slider to wipe from one to the other
Diff	The capture, with the changed pixels highlighted over it. For a changed snapshot

An unchanged snapshot stores no image of its own, so you see its baseline.

Approving and rejecting

Members and above can Approve or Reject. Guests can open Visual Testing and look at every comparison, but not decide.

• Approve makes this run's capture the new baseline for its branch. Every later run on that branch compares against it. See Branches for which baselines that touches.
• Reject keeps the old baseline and records why, if you give a reason (up to 500 characters). The next run compares against the same baseline as before.
• A rejected snapshot can be approved later. An approval cannot be undone. If an approved picture turns out to be wrong, approve a correct one from a later run.
• An older capture cannot replace a newer baseline. If a newer run's capture of the same snapshot has already been approved, approving this one is refused and the review says so: review it in the newer run instead.
• Once retention has removed a snapshot's images, it can no longer be approved.

Only changed and new snapshots are reviewed. An unchanged one has nothing to decide, and a failed one has nothing to approve.

Approve all

Approve all approves every snapshot waiting in the run, up to 500 at once. Any it cannot approve, such as one whose baseline is already newer, are left as they are, and the page says how many and why.

It is offered on larger screens only. On a phone you can read a run and approve or reject one snapshot at a time.

Approve all only ever means what it says: the request has to state that it wants every waiting snapshot, and one that names nothing is refused. The button does this for you. Anything else that calls the approve endpoint, or an assistant using the AI connector, has to either list the snapshots to approve or say it means all of them.

:::note Recently changed The approve endpoint used to approve every waiting snapshot when it was sent an empty request, an empty list or an unreadable list of ids. Those are refused now, because a call that names nothing should not change what every later run is compared against. Approve all in the app is unchanged, except that a page opened before the release needs a reload before the button works again. :::

In the activity log

Each decision is recorded in the activity log as Approved visual changes or Rejected a visual change, against the automation run, and the entry links to that run's Visual Testing page. An Approve all is one entry.

Branches

Each branch keeps its own baselines, so two pull requests can change the same page without overwriting each other.

The CLI reads the branch from CI: GitHub Actions, GitLab, CircleCI, Bitbucket, Azure Pipelines and Jenkins all say which branch they are building, and on a pull request it takes the source branch. --branch sets it yourself. A tag build has no branch, so it is treated like the default branch.

The default branch is the one --default-branch names, or the one CI reports where it does (GitLab, and GitHub's event). Otherwise main and master count as default.

The run is on	It compares against	Approving changes
The default branch, or no branch	The default branch's baselines	The default branch's baselines
A feature branch with its own baseline for that snapshot	That branch's baseline	That branch's baselines only
A feature branch without one	The default branch's baseline	That branch's baselines only

The default branch's baselines change only through approvals on the default branch, or on runs with no branch. A branch's approvals stay on that branch, so the first run on the default branch after a merge shows the change again, and someone approves it there.

Tolerances

Two captures of the same page are rarely identical to the pixel. The tolerances decide how much difference still counts as unchanged. They have the same names as Playwright's toHaveScreenshot, and are set per snapshot in the helper's options:

Option	Default	What it does
threshold	0.1	How far a pixel's colour may move before the pixel counts as changed, from 0 (any difference) to 1
maxDiffPixels	0	How many changed pixels still count as unchanged
maxDiffPixelRatio	0	The share of the image's pixels, from 0 to 1, that may change and still count as unchanged

• Pixels that differ only because of anti-aliasing, the smoothing along the edges of text and shapes, are ignored.
• When both maxDiffPixels and maxDiffPixelRatio are set, staying within either one is enough.
• A change in the image's size is always a change, whatever the tolerances.

Before raising a tolerance to quiet a snapshot that differs on every run, try mask on the part that moves. A clock painted over is still a clock; a 2% tolerance also hides a 2% regression.

Results

Each snapshot gets one result when it arrives, and it never changes. What a person decides is recorded beside it, so an approved change still reads as a change that somebody approved.

Result	Meaning	What is stored
unchanged	Identical to its baseline, or within the tolerances	Nothing. The review shows the baseline
changed	Differs from its baseline	The capture and a diff
new	There is no baseline yet	The capture
failed	Could not be compared. The reason is shown	Nothing

A snapshot fails when it is not a PNG, when it is over the size or pixel limits below, or when the organization has reached its storage allowance.

Limits

	Limit
Image format	PNG only
One image	20 MB, and 16,777,216 pixels (4096 × 4096)
One request	100 snapshots. The CLI sends them in batches for you
One run	2,000 snapshots
A snapshot's name	200 characters

Only PNG, because JPEG and WebP add compression noise that a pixel comparison reads as change.

Sending snapshots to the same run again is safe. Sekee answers with the result each had the first time, and what was decided since, and stores nothing new, so a retried CI step never counts twice.

Retention

• Captures and diffs of changed and new snapshots follow your plan's artifact retention: 90 days on Business, counted from the day they arrive. After that the review says the images were removed by retention. The snapshot's result, its review and the run's counts stay.
• The default branch's current baselines are kept for as long as you are subscribed. They are what every run compares against.
• A feature branch's baselines are removed once no run has compared against them for a retention period. That branch then falls back to the default branch's baselines.
• A baseline replaced by a newer approval is kept for a retention period.
• An unchanged snapshot stores no image at all.

All of it counts toward the organization's storage allowance.

Deleting a project deletes its snapshots and baselines. The organization export includes snapshots and baselines as records, never the image files.

Troubleshooting

Every snapshot changed after a runner change

The machine that renders the pages changed: a new runner image, a new Playwright Docker tag, or different fonts installed. The platform is still linux, so the snapshots are compared, and every one differs a little. Pin the Docker image so it only changes when you change it. When you did change it on purpose, run once on the default branch and use Approve all.

Every snapshot is new

The platform, viewport or Playwright project name is part of a snapshot's identity. A run from a Mac never compares against baselines captured on Linux CI, so it starts baselines of its own, all new. The same happens when a viewport changes or a Playwright project is renamed. Capture in the same Docker image wherever you run the suite.

"A newer capture is already the baseline"

You are approving a snapshot from an older run, and a newer run's capture of the same snapshot on the same branch has already been approved. Approving the older one would roll the baseline back, so Sekee refuses. Review it in the newer run instead.

The upload answers 400 with RunClosed

The run is completed and more than 24 hours old, so it no longer takes snapshots. Start a new run and send them there. The CLI stops at the first such answer rather than sending the rest.

Snapshots failed because of storage

The CLI prints a warning that the organization is over its storage allowance, and the snapshots it could not keep are failed. Unchanged snapshots cost nothing, but every changed and new capture, and every diff, is stored. See Plans and billing. A failed snapshot stays failed in its run, so once there is room, send the snapshots again in a new run.

The job stays red after approving

Something in the run still fails the step. Open the run and look at three tabs:

• Needs review: a change nobody has approved yet. Approve all leaves out any it cannot approve, and says why.
• Rejected: a rejected change keeps failing until a run captures the page fixed.
• Failed: a snapshot that could not be compared fails in that run whatever is approved. Fix the cause and start a new run.

See Failing the build on a change.

"No snapshot directory" or "No visual snapshots"

snapshots found nothing to send and exited 0. Either the tests captured no snapshots, or the helper wrote them somewhere else: pass the same --dir you gave the helper's dir option or SEKEE_SNAPSHOTS_DIR.