GitHub Actions & GitLab CI Gates for Map Visual Tests

A visual test that captures a diff but never fails the pipeline is documentation, not a gate. The job of a CI gate is to make a bad render un-mergeable: when the changed-pixel ratio for any view crosses its per-zoom budget, the pipeline exits non-zero, the merge is blocked, and the diff images are one click away in the run’s artifacts. Wiring that into GitHub Actions or GitLab CI for a map suite is deceptively fiddly, because the thing you are gating on — pixels out of a WebGL rasterizer — is exquisitely sensitive to the environment. The runner image, the installed font faces, the GL backend, and even the tile bytes have to match the environment that produced the baselines, or every pull request lights up with false failures that have nothing to do with the code under review. This page is a concrete, step-by-step walkthrough of a gate that enforces thresholds for real, on both platforms, without that drift.

This page extends the CI/CD Integration & Visual Test Operations parent guide, which frames why visual regression only earns its keep when it gates a deploy. Here we narrow to one question: what is the exact YAML, on GitHub Actions and GitLab CI, that turns a capture-and-diff run into a hard merge gate — and how do you keep it deterministic across runners.

What a “gate” actually enforces #

A gate is a pipeline step whose exit code is load-bearing. The diff stage compares each current capture against its blessed baseline, computes a changed-pixel ratio, compares that ratio to a budget, and — this is the part teams get wrong — returns a non-zero exit code when any view is over budget. CI platforms map process exit status directly onto job status: a zero exit passes, anything else fails, and a failed required job blocks the merge. Everything else on this page exists to make that single exit code trustworthy.

The changed-pixel ratio is the primitive the budget is expressed against. For a capture of width $W$ and height $H$ device pixels, with pixelmatch reporting $c$ changed pixels, the ratio is

$$\rho =\frac{c}{W\cdot H}$$

and the gate fails the view when $\rho >{\rho }_{\max }(z)$, where the budget ${\rho }_{\max }$ is a function of zoom level $z$ — tighter at street level where a dropped label matters, looser at continental zoom where generalized coastlines and gradients legitimately dither. Those per-zoom budgets come straight from the tuning work in Dynamic Threshold Configuration; the gate consumes them, it does not invent them. The precise mechanics of computing and asserting $\rho$ inside a GitHub Actions step — including how to surface a per-view summary and annotate the failing check — are covered in enforcing pixel diff thresholds in GitHub Actions for maps.

Two properties make a gate more than a glorified if:

• Determinism. The same commit must produce the same $\rho$ on every runner. If the runner image, fonts, or GL backend drift from the baseline environment, $\rho$ moves for reasons unrelated to the diff, and the gate becomes noise the team learns to re-run past.
• Fail-closed exit semantics. The diff step must propagate a non-zero exit through every shell wrapper, npm script, and set -e boundary between it and the job runner. A diff that logs “FAIL” but exits 0 is the single most common way a gate silently stops gating.

Architecture of a gated map pipeline #

The pipeline is a fixed sequence of stages, and the order is not negotiable — each stage depends on an invariant the previous one established. Checkout brings the code and the manifests. Installing pinned fonts and browsers reconstructs the exact rendering environment. Pulling baselines from object storage brings the goldens the diff will compare against. Capture produces the current frames under that frozen environment. Diff computes $\rho$ per view and compares it to the per-zoom budget. Pass or fail is the exit code. On failure, the diff artifacts upload so a human can see what moved.

The environment-reconstruction stage is the one that separates a map gate from a generic screenshot gate. The reproducibility contract is set once, in a container image, and both CI platforms consume the same image so local, GitHub, and GitLab render identically. That contract — pinned base image, pinned font packages, forced software GL backend — is developed in full in Containerized Rendering Environments for Map Tests; the gate assumes it and pins the same digest. Baselines themselves live outside the repo, keyed to the style hash that produced them, as Baseline Management for Tile Servers lays out; the pipeline pulls them at the start of every run rather than committing binary PNGs.

Step-by-step: a gate on GitHub Actions #

The following builds the pipeline stage by stage. Each numbered step maps to one or two YAML keys, so you can assemble the whole workflow from the fragments.

1. Pin the runner image to a digest, not a floating tag. ubuntu-latest silently rolls forward and takes its GPU stack and font set with it. Run inside the same container image that generated the baselines, referenced by immutable digest so a rebuild upstream cannot change your pixels.

   jobs:
     map-visual:
       runs-on: ubuntu-22.04
       container:
         image: ghcr.io/acme/map-visual@sha256:9f2c…e1
       env:
         TZ: UTC
         LANG: en_US.UTF-8
   

2. Reconstruct fonts and the GL backend even inside the image. If you cannot run a prebuilt image, install the exact font packages and force software GL at the step level. A missing font face substitutes a fallback and shifts every label by a subpixel — a full-frame diff with no code change.

       steps:
         - uses: actions/checkout@v4
         - name: Install pinned fonts
           run: |
             apt-get update
             apt-get install -y fonts-noto-core=20201225-1build1 fonts-noto-cjk
         - name: Install browsers
           run: npx playwright install --with-deps chromium
   

   The capture itself launches Chromium with --use-gl=angle --use-angle=swiftshader so WebGL rasterizes on the CPU identically across nodes, regardless of the host GPU.

3. Pull baselines from object storage keyed by style hash. Baselines are not in Git. Resolve the style hash for the current commit, then fetch exactly that baseline set from S3 (or GCS) so the diff compares against the goldens for this style, not last month’s.

         - name: Pull baselines
           env:
             AWS_ACCESS_KEY_ID: ${{ secrets.BASELINE_RO_KEY }}
             AWS_SECRET_ACCESS_KEY: ${{ secrets.BASELINE_RO_SECRET }}
           run: |
             STYLE_HASH=$(node scripts/style-hash.js)
             aws s3 sync "s3://acme-map-baselines/${STYLE_HASH}/" baseline/ --only-show-errors
   

4. Run capture and diff as one step whose exit code is the gate. The test runner captures each view under the frozen environment, computes $\rho$ per view, compares it against the per-zoom budget, and exits non-zero if any view is over. Do not swallow the exit code with || true or a trailing ; echo done.

         - name: Capture and diff (the gate)
           run: npm run test:visual
   

   Inside test:visual, the assertion is explicit — the process must actually throw or process.exit(1):

   let failed = 0;
   for (const view of views) {
     const changed = pixelmatch(base[view].data, cur[view].data, diff[view].data,
       W, H, { threshold: 0.1, includeAA: false });
     const rho = changed / (W * H);
     const budget = zoomBudget(view.zoom);      // per-zoom ρmax
     if (rho > budget) {
       fs.writeFileSync(`diff/${view.id}.png`, PNG.sync.write(diff[view]));
       console.error(`FAIL ${view.id}: ρ=${(rho * 100).toFixed(3)}% > ${(budget * 100).toFixed(3)}%`);
       failed++;
     }
   }
   if (failed > 0) process.exit(1);            // ← this line is the gate
   

5. Upload diff artifacts only on failure. A passing run should not spend storage or minutes uploading unchanged frames. Gate the upload on failure() and set a retention window so diffs expire.

         - name: Upload diffs
           if: failure()
           uses: actions/upload-artifact@v4
           with:
             name: visual-diffs-${{ github.sha }}
             path: diff/
             retention-days: 14
   

6. Make the job a required status check. In branch protection, mark map-visual as required. Now the failed exit code from step 4 blocks the merge button; the check status is what enforces the gate at the PR level, not the log output.

Step-by-step: the same gate on GitLab CI #

GitLab expresses the identical pipeline with different primitives. The container is the job image, artifacts are declared with when: on_failure, and the gate is enforced through merge request rules rather than a status check API.

map-visual:
  image: registry.example.com/acme/map-visual@sha256:9f2c…e1
  stage: test
  variables:
    TZ: UTC
    LANG: en_US.UTF-8
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  script:
    - node scripts/style-hash.js > .style-hash
    - aws s3 sync "s3://acme-map-baselines/$(cat .style-hash)/" baseline/ --only-show-errors
    - npm run test:visual        # exits non-zero when any view is over budget
  artifacts:
    when: on_failure             # upload diffs only when the gate fails
    paths:
      - diff/
    expire_in: 14 days
  retry:
    max: 1
    when: runner_system_failure  # retry infra flakes, never a real diff

Three GitLab-specific points matter for a map gate. First, artifacts: when: on_failure is the direct analogue of GitHub’s if: failure() — it keeps passing runs cheap. Second, the retry block is scoped to runner_system_failure only; retrying on script_failure would paper over a genuine visual regression, which is exactly the flake-masking anti-pattern the Cross-Browser Baseline Matrix guide warns against when you run the same gate across several engines. Third, the gate is made blocking by enabling “Pipelines must succeed” in the project’s merge request settings, so a non-zero test:visual exit prevents the merge just as a required check does on GitHub.

The exit-code contract is identical on both platforms: npm run test:visual is the gate, and it must return non-zero when any view is over budget. If your runner suite is large enough that a single job is too slow, split the capture across shards — the sharding strategy, and how to merge per-shard results back into one gate verdict, is the subject of parallelizing map screenshot tests across CI shards.

Branch-based baseline blessing #

A gate is only half the workflow; the other half is how a legitimate visual change gets blessed into the new baseline. The safe pattern is branch-based. On feature branches and merge requests, the pipeline runs in gate mode: it pulls the current baselines and fails on any over-budget view. On the protected default branch — after a change has been reviewed and merged — a separate bless job runs in update mode: it captures fresh frames and pushes them to object storage under the new style hash, becoming the baseline every subsequent branch compares against.

bless-baselines:
  image: registry.example.com/acme/map-visual@sha256:9f2c…e1
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'   # only on the protected branch
  script:
    - node scripts/style-hash.js > .style-hash
    - npm run capture:baseline            # capture, no diff
    - aws s3 sync baseline/ "s3://acme-map-baselines/$(cat .style-hash)/" --only-show-errors

Keying the upload to the style hash is what prevents the classic blessing race: two branches that both change the style produce two different hashes, so their baselines land in different prefixes and never clobber each other. Blessing on the protected branch only — never from an unreviewed branch — means a baseline can only be updated by code that has already passed review, closing the loop between the gate and the goldens it enforces against.

Cross-environment considerations #

The gate’s verdict is only portable if the environment is. Three axes drift most often between a developer laptop, a GitHub runner, and a GitLab runner:

• Base image digest. Pin by @sha256: digest, not tag. playwright:v1.44-jammy can be rebuilt with new fonts under the same tag; the digest cannot. Both platforms reference the same digest so all three environments share one GL stack and font set.
• GL backend. Force --use-gl=angle --use-angle=swiftshader everywhere. A host with a real GPU (a developer’s machine) will otherwise dither gradients and anti-alias labels differently from a headless CI node, and every locally-blessed baseline will fail in CI.
• DPR and viewport. Lock deviceScaleFactor and viewport dimensions in the browser context. A DPR-2 baseline blessed on a retina laptop doubles the tile count and changes anti-aliasing versus a DPR-1 runner; never mix the two in one baseline set.

When the divergence is between browser engines rather than runners — Chromium versus WebKit versus Firefox rendering the same MapLibre style — the answer is a separate baseline per engine, not a looser threshold. That matrix is maintained as described in the Cross-Browser Baseline Matrix guide, and each engine’s job carries its own per-zoom budgets.

Threshold & parameter reference #

Starting values for the gate’s knobs. Tune the changed-pixel budgets against the per-zoom tables from Dynamic Threshold Configuration; the CI-specific knobs (retry, retention, concurrency) balance signal against runner cost.

On GitHub, the concurrency knob is a top-level block that cancels a superseded run when a new commit lands on the same branch, so you never pay for a diff against an abandoned commit:

concurrency:
  group: map-visual-${{ github.ref }}
  cancel-in-progress: true

Common pitfalls #

Passes locally, fails only in CI

Root cause: environment drift — the runner’s GL backend, font packages, base image digest, or DPR differs from the machine that blessed the baseline, so $\rho$ moves for reasons unrelated to the code. Diagnose: re-bless a baseline on the runner image itself (via docker run on the pinned digest) and diff it against the laptop-blessed one; a non-empty diff with no code change confirms drift, not regression. Fix: pin the base image by @sha256: digest, install fonts by exact package version, force --use-gl=angle --use-angle=swiftshader, and lock DPR and viewport. Reconstruct the same contract described in Containerized Rendering Environments for Map Tests on both platforms.

A warm cache lets a run skip hydration and pass a broken frame

Root cause: HTTP caching or a persisted CI cache served tiles, glyphs, or sprites from a previous run, so the capture never exercised the hydration path that would have exposed a missing asset — the frame looks fine because it was assembled from stale bytes. Diagnose: clear the CI cache and disable tile-endpoint caching, then re-run; if the frame now differs, the cache was masking the fault. Fix: serve tiles, sprites, and style JSON from local fixtures inside the job, disable HTTP caching on tile endpoints, and never cache the baseline/ or current/ directories between runs.

The diff step logs FAIL but the job goes green

Root cause: the non-zero exit never reached the runner — the diff was wrapped in || true, a trailing command overwrote $?, a set +e swallowed it, or the test framework reported failures without exiting non-zero. Diagnose: run npm run test:visual; echo "exit=$?" in the step and confirm the code is non-zero on a known-bad view. Fix: ensure the diff script calls process.exit(1) (or throws through the framework’s reporter), remove any || true, and keep the gate as the last command in the step so its exit code is the step’s exit code.

Two branches bless baselines and clobber each other

Root cause: baselines were written to a shared, unversioned prefix, so whichever branch’s bless job ran last overwrote the other’s goldens, and the loser’s next gate run failed against the wrong baseline. Diagnose: compare the style hash in a failing run’s log against the prefix actually present in object storage; a mismatch means a race. Fix: key every baseline upload to the style hash of the commit that produced it, and bless only from the protected branch after review, so unreviewed branches can never write goldens.

Diff artifacts upload on every run and blow up storage

Root cause: the upload step is unconditional, so passing runs archive unchanged frames, and with no retention window the bucket grows without bound. Diagnose: check artifact storage growth against pass rate; if passing runs carry artifacts, the condition is missing. Fix: gate the upload on if: failure() (GitHub) or when: on_failure (GitLab), set retention-days / expire_in, and upload only the diff/ directory, not the full baseline and current sets.

Frequently asked questions #

How does the diff step actually fail the pipeline?

CI platforms map process exit status onto job status: a zero exit passes, non-zero fails. The diff script must call process.exit(1) (or throw through the test reporter) when any view’s changed-pixel ratio exceeds its per-zoom budget, and that command must be the last thing in the step with no || true swallowing its code. On GitHub the failed required check blocks the merge; on GitLab “Pipelines must succeed” does the same for the merge request.

Where should baselines live — in the repo or object storage?

In object storage, keyed by the style hash that produced them. Binary PNGs in Git bloat the repository and drift from the styles they belong to. The pipeline pulls exactly the baseline set for the current style hash at the start of each run and blesses new baselines only from the protected branch, as Baseline Management for Tile Servers details.

Should I retry a failing visual job automatically?

Only for infrastructure failures — a runner that died or lost the network — never for a real diff. Scope GitHub reruns and GitLab’s retry: when: runner_system_failure to infra causes. Auto-retrying on a script_failure re-runs the diff and can pass a genuinely regressed view on a lucky second attempt, which defeats the gate. Fix flake at its source (hydration, GL backend) rather than retrying past it.

Related #

• Up to the parent guide CI/CD Integration & Visual Test Operations — why visual regression only earns its keep when it gates a deploy.
• Enforcing pixel diff thresholds in GitHub Actions for maps — computing and asserting the changed-pixel ratio inside a step.
• Parallelizing map screenshot tests across CI shards — splitting capture across shards and merging into one verdict.
• Cross-Browser Baseline Matrix — a separate baseline per engine instead of a looser threshold.
• Containerized Rendering Environments for Map Tests and Baseline Management for Tile Servers — the pinned environment and object-storage goldens the gate depends on.
• Dynamic Threshold Configuration — where the per-zoom budgets the gate enforces come from.

← Back to CI/CD Integration & Visual Test Operations