Enforcing Pixel-Diff Thresholds in GitHub Actions for Maps
A visual regression suite only protects a map when a bad frame actually stops the merge. The missing piece is usually not the diff — it is the wiring that turns a changed-pixel count into a non-zero exit code, surfaces that exit code as a required status check, and refuses the pull request until a human looks at the artifacts. This guide builds that job end to end in GitHub Actions: a pinned ubuntu-22.04 runner, deterministic fonts and Node, baselines pulled from an artifact, a pixelmatch diff computed per screenshot, and a script that compares each map’s changed-pixel ratio against a per-zoom budget and exits 1 the moment any zoom level breaches it.
This is one task inside GitHub Actions & GitLab CI Gates for Map Visual Tests, the guide that covers wiring map visual checks into pipeline gates, and it sits under the broader area described in CI/CD Integration & Visual Test Operations. It assumes you already capture deterministic frames; the gate here consumes those frames rather than producing them.
Prerequisites
Step-by-step procedure
1. Pin the runner, fonts, and Node
Anti-aliased labels and glyph hinting shift by a pixel or two when the font stack or GL backend changes, and that drift alone can blow a tight budget. Pin the runner image to ubuntu-22.04 (never ubuntu-latest, which silently rolls forward), install a fixed font package, and pin Node to a single minor. This is the same environment-determinism discipline that the containerized rendering environments guide argues for; the gate below inherits its stability from it.
# .github/workflows/map-visual-gate.yml
name: Map Visual Gate
on:
pull_request:
branches: [main]
jobs:
pixel-diff-gate:
runs-on: ubuntu-22.04 # pinned, never ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Pin fonts for deterministic label rendering
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
fonts-noto-core fonts-noto-cjk fontconfig
sudo fc-cache -f
- name: Pin Node
uses: actions/setup-node@v4
with:
node-version: '20.11.1'
cache: npm
- name: Install dependencies
run: npm ci
2. Pull the approved baselines
Keep baselines out of the working tree change set so a PR cannot silently overwrite them. Download the last approved set as an artifact (published by the main-branch build) into a sibling directory. If you instead store baselines with Git LFS, swap this step for an lfs checkout — the rest of the job is identical.
- name: Download approved baselines
uses: actions/download-artifact@v4
with:
name: map-baselines
path: test/__screenshots__/baseline
# baselines are published by the main-branch build, not the PR
3. Run the capture that produces current frames
Run your existing screenshot step so the current/ directory is populated for this commit. The gate treats capture as a black box: its only contract is one PNG per map × zoom case, named identically to its baseline so the diff script can pair them.
- name: Capture current map frames
run: npm run test:visual:capture
# writes test/__screenshots__/current/<name>.z<zoom>.png
4. Compute the changed-pixel ratio and gate on the per-zoom budget
This is the heart of the job. The script below pairs each current frame with its baseline, runs pixelmatch to count changed pixels, divides by the total to get a ratio, parses the zoom level out of the filename, and compares the ratio to that zoom’s budget. Any breach pushes a record onto a failures array; at the end, a non-empty array means process.exit(1), which fails the step and therefore the check.
The budget widens with zoom on purpose: high zoom shows more anti-aliased label edges and hillshade gradients, so a fixed global threshold either flakes at z16 or goes blind at z4. The reasoning behind these numbers lives in the per-zoom diff threshold tables for slippy maps guide.
// scripts/pixel-diff-gate.mjs
import { readdirSync, readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { join, basename } from 'node:path';
import { PNG } from 'pngjs';
import pixelmatch from 'pixelmatch';
const CURRENT = 'test/__screenshots__/current';
const BASELINE = 'test/__screenshots__/baseline';
const DIFF_OUT = 'test/__screenshots__/diff';
// Changed-pixel ratio budget per zoom level. Ratios, not percentages.
const ZOOM_BUDGET = {
4: 0.0015, 8: 0.0020, 10: 0.0030,
12: 0.0040, 14: 0.0060, 16: 0.0090,
};
const DEFAULT_BUDGET = 0.0040;
// pixelmatch per-pixel colour tolerance; below this, a pixel is "unchanged".
const MATCH_THRESHOLD = 0.1;
const zoomOf = (name) => {
const m = name.match(/\.z(\d+)\.png$/);
return m ? Number(m[1]) : null;
};
const budgetFor = (zoom) => ZOOM_BUDGET[zoom] ?? DEFAULT_BUDGET;
mkdirSync(DIFF_OUT, { recursive: true });
const frames = readdirSync(CURRENT).filter((f) => f.endsWith('.png'));
const results = [];
const failures = [];
for (const file of frames) {
const zoom = zoomOf(file);
const budget = budgetFor(zoom);
const cur = PNG.sync.read(readFileSync(join(CURRENT, file)));
const base = PNG.sync.read(readFileSync(join(BASELINE, file)));
if (cur.width !== base.width || cur.height !== base.height) {
failures.push({ file, zoom, reason: 'dimension-mismatch' });
continue;
}
const { width, height } = cur;
const diff = new PNG({ width, height });
const changed = pixelmatch(
base.data, cur.data, diff.data, width, height,
{ threshold: MATCH_THRESHOLD, includeAA: false },
);
const ratio = changed / (width * height);
const breached = ratio > budget;
if (breached) {
writeFileSync(join(DIFF_OUT, `diff-${file}`), PNG.sync.write(diff));
failures.push({ file, zoom, ratio, budget });
}
results.push({ file, zoom, changed, ratio, budget, breached });
}
// Machine-readable summary for the PR-comment step.
writeFileSync('pixel-diff-report.json',
JSON.stringify({ results, failures }, null, 2));
for (const r of results) {
const pct = (r.ratio * 100).toFixed(3);
const cap = (r.budget * 100).toFixed(3);
const flag = r.breached ? 'FAIL' : 'ok';
console.log(`[${flag}] ${basename(r.file)} z${r.zoom} ${pct}% / ${cap}%`);
}
if (failures.length > 0) {
console.error(`\n${failures.length} frame(s) exceeded the per-zoom budget.`);
process.exit(1); // non-zero exit fails the GitHub Actions step
}
console.log('\nAll frames within budget.');
Wire it into the workflow as its own step. Because the script calls process.exit(1) on any breach, this step is the gate — everything after it runs only on failure (thanks to if: failure()).
- name: Enforce per-zoom pixel-diff budget
id: gate
run: node scripts/pixel-diff-gate.mjs
5. Upload diff artifacts when the gate fails
A red check with no evidence forces the author to reproduce locally. Upload the generated diff PNGs and the JSON report only when the gate step failed, so a reviewer can open the exact frames that breached without re-running anything.
- name: Upload diff artifacts on failure
if: failure() && steps.gate.outcome == 'failure'
uses: actions/upload-artifact@v4
with:
name: pixel-diff-failures
path: |
test/__screenshots__/diff/
pixel-diff-report.json
retention-days: 14
6. Comment the ratio back onto the pull request
Post the per-zoom ratios inline so the failure is legible from the PR timeline. Read pixel-diff-report.json and render a short table via actions/github-script. Grant the job pull-requests: write so the comment can be created.
- name: Comment diff ratios on the PR
if: failure() && steps.gate.outcome == 'failure'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const { failures } = JSON.parse(
fs.readFileSync('pixel-diff-report.json', 'utf8'));
const rows = failures.map(f =>
`| \`${f.file}\` | z${f.zoom} | ${(f.ratio*100).toFixed(3)}% | ${(f.budget*100).toFixed(3)}% |`
).join('\n');
const body = [
'### Map visual gate failed',
'',
'| Frame | Zoom | Changed | Budget |',
'|---|---|---|---|',
rows,
'',
'Diff PNGs are in the **pixel-diff-failures** artifact.',
].join('\n');
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body,
});
7. Make the check required
The exit code only blocks a merge if branch protection says it must. In the repository’s branch-protection rule for main, enable Require status checks to pass before merging and add the pixel-diff-gate job by name. Until you do, the red check is advisory and anyone can merge past it.
Verification
Confirm the gate blocks a real regression, not just a green baseline:
If the negative control merges cleanly, the check is not marked required yet — return to step 7.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Gate is red but the merge button stays enabled | The status check is not listed as required in branch protection | Add pixel-diff-gate to the required checks for main (step 7); the exit code alone never blocks a merge |
Every frame fails with dimension-mismatch |
Current capture uses a different device pixel ratio or viewport than the baseline set | Pin viewport and DPR in the capture step so both sets share width and height, then rebaseline once from the pinned runner |
| High-zoom frames flake red on unrelated PRs | A single global threshold is too tight for anti-aliased labels at z14–z16 | Widen only the high-zoom rows of ZOOM_BUDGET using the per-zoom diff threshold tables for slippy maps, and keep includeAA: false |
Frequently asked questions
Why gate on a changed-pixel ratio instead of an absolute pixel count?
An absolute count does not travel across viewport sizes: 5,000 changed pixels is a rounding error on a 4K frame and a catastrophe on a thumbnail. A ratio — changed / (width × height) — normalizes the budget so the same number is meaningful for every map case, and it lets you reason about “fraction of the frame that moved” directly. Keep the budget as a ratio in ZOOM_BUDGET and only convert to a percentage for display.
Should the gate run per shard or after a merge step?
Run the diff inside each shard that captured frames, so a breach fails fast and localizes to the runner that produced it, then require all shards to pass. Splitting capture across runners is covered in parallelizing map screenshot tests across CI shards; the gate script here is unchanged — each shard runs it over its own subset of frames.
How do I stop font drift from eating the whole budget?
Font rendering is the most common source of “changed” pixels that are not real regressions. Pin the font package and refresh the cache in step 1, keep the GL backend fixed, and run capture inside a pinned image as described in containerized rendering environments. With includeAA: false, pixelmatch already discounts pure anti-aliasing differences, so most residual noise disappears once the environment is stable.
Why does process.exit(1) fail the job when a thrown error would too?
Both fail the step, but they differ in intent and output. A breach is an expected outcome the gate is designed to report, so it prints a clean per-frame table and exits 1 deliberately; a thrown error is an unexpected fault (missing baseline, unreadable PNG) that should surface a stack trace. Keeping breaches on process.exit(1) and letting genuine faults throw makes the logs immediately tell you which of the two happened.
Related
- Up to the parent guide GitHub Actions & GitLab CI Gates for Map Visual Tests, and the broader area CI/CD Integration & Visual Test Operations.
- Parallelizing map screenshot tests across CI shards — running this gate inside each shard.
- Per-zoom diff threshold tables for slippy maps — where the budget numbers come from.
- Containerized rendering environments — pinning fonts and the GL backend so drift never spends the budget.