Parallelizing Map Screenshot Tests Across CI Shards
A map screenshot suite grows faster than a normal end-to-end suite: every basemap style, zoom level, engine, and viewport multiplies into another baseline, and each capture drags a headless browser plus a GPU-backed WebGL context behind it. Left on one runner, a few hundred map states turn a pull request into a twenty-minute wait, and the gate that was meant to protect cartography quietly becomes the thing people route around. This guide walks through splitting that suite across parallel CI shards so wall-clock time falls roughly linearly with runner count — while keeping the diff verdict a single, trustworthy check on the pull request rather than a scatter of per-shard results nobody reads.
This task sits inside GitHub Actions & GitLab CI Gates for Map Visual Tests, the guide covering how a map diff becomes a required status check, and under the broader area of CI/CD Integration & Visual Test Operations. It assumes you already have a working single-runner gate — the sharding here is a scaling move, not a first setup.
Prerequisites
Step-by-step procedure
1. Choose a shard key that keeps each shard self-contained
A shard is only cheap if it needs nothing from its siblings. The two shard keys that hold up for map suites are by view (each shard owns a disjoint set of routes, styles, or camera states) and by engine (each shard owns one renderer — Chromium, WebKit, Firefox — so its browser and baselines are homogeneous). Prefer a view-based split when one engine dominates and the suite is large; prefer an engine-based split when the same views run across several engines and you already maintain per-engine baselines. The anti-pattern is splitting mid-way through a page’s states so two shards both load the same style and both need the same tile fixtures — that double-provisions memory and risks double-capturing a shared baseline.
// playwright.config.ts — group specs so a shard key maps to whole, disjoint units
export default defineConfig({
projects: [
{ name: 'chromium-basemaps', testMatch: /basemap\..*\.spec\.ts/ },
{ name: 'chromium-overlays', testMatch: /overlay\..*\.spec\.ts/ },
{ name: 'webkit-basemaps', testMatch: /basemap\..*\.spec\.ts/, use: { browserName: 'webkit' } },
],
});
2. Split the run with Playwright --shard=i/n
Playwright’s built-in sharding hashes the test list into n buckets and runs bucket i. It is deterministic across runs given the same test set, so shard 3 always owns the same specs — which is what lets each shard restore only the fixtures it needs. Pass the index and total straight through from the CI matrix.
# each runner executes exactly one of these, i in 1..n
npx playwright test --shard="${SHARD_INDEX}/${SHARD_TOTAL}"
For balance, Playwright shards by test count, not by cost. A map capture that waits on a cold tile fetch is far heavier than one hitting a warm fixture, so equal test counts do not mean equal wall-clock. Step 6 covers rebalancing when one shard runs long.
3. Drive the shards from a GitHub Actions matrix
The matrix fans one job definition out into n parallel jobs, injecting the shard index into each. Pin the browser image, restore fixtures, and pass matrix.shard/matrix.total into the command. fail-fast: false is deliberate — you want every shard to finish and report, not have the first red shard cancel the rest and hide the full diff picture.
name: map-visual
on: pull_request
jobs:
screenshots:
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.44.0-jammy # pinned: fonts + GL fixed
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4]
total: [4]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 20, cache: npm }
- run: npm ci
- name: Restore tile fixtures for this shard
uses: actions/cache@v4
with:
path: fixtures/tiles
key: tiles-${{ hashFiles('fixtures/manifest.lock') }}
- name: Run shard
env:
SHARD_INDEX: ${{ matrix.shard }}
SHARD_TOTAL: ${{ matrix.total }}
run: npx playwright test --shard="$SHARD_INDEX/$SHARD_TOTAL"
- name: Upload shard report
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: blob-report-${{ matrix.shard }}
path: blob-report
retention-days: 1
Each shard emits a Playwright blob report — a machine-mergeable partial result — rather than an HTML report, because HTML reports cannot be combined. That blob is the unit step 5 stitches back together.
4. Provision each shard’s fixtures and pinned browser identically
Sharding multiplies your environment-determinism problem by n: a pixel that renders differently on shard 2 than shard 1 produces a false diff that depends on which shard happened to own the spec. Every runner must therefore present byte-identical rendering conditions. Pin the browser through the container image (step 3), force the software GL backend, and serve tiles from fixtures so no shard races a live CDN.
# identical on every shard — software WebGL so GPU variance can't leak per node
export PW_CHROMIUM_ARGS="--use-gl=angle --use-angle=swiftshader --force-device-scale-factor=1"
# point the renderer at a per-shard local fixture server, not a shared live endpoint
export TILE_BASE_URL="http://127.0.0.1:8080/tiles"
Restore fixtures with a content-addressed cache key (the hashFiles('fixtures/manifest.lock') above) so a fixture change busts every shard’s cache in lockstep. If one shard silently ran against stale tiles, its baselines would drift out from under the others.
5. Merge the per-shard blob reports into one PR check
The whole point of sharding is defeated if reviewers must open four separate reports. Collect every shard’s blob artifact in a dependent job and let Playwright merge them into a single HTML report and, more importantly, a single pass/fail exit code that becomes the one required status check on the pull request.
merge-report:
needs: screenshots
if: ${{ !cancelled() }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 20, cache: npm }
- run: npm ci
- name: Download all shard blob reports
uses: actions/download-artifact@v4
with:
pattern: blob-report-*
path: all-blobs
merge-multiple: true
- name: Merge into one report
run: npx playwright merge-reports --reporter=html ./all-blobs
- name: Publish merged diff report
uses: actions/upload-artifact@v4
with: { name: visual-report, path: playwright-report }
Because merge-reports re-derives the aggregate status from the combined blobs, a single failing diff in any shard fails the merged check — the gate stays as strict as the single-runner version it replaced. This is the same threshold enforcement described in enforcing pixel diff thresholds in GitHub Actions for maps, now applied once over the merged set instead of per shard.
6. Balance shard load so the slowest shard bounds the run
Wall-clock for a sharded run equals the slowest shard, not the average. Headless browsers are memory-heavy — each Chromium context with a live WebGL canvas can hold hundreds of megabytes — so a shard that packs the tile-heavy, high-zoom captures both runs longest and risks the runner’s out-of-memory killer. Measure per-shard duration from the merged report, then move heavy specs across the key boundary or add a shard. If one engine (WebKit) is consistently slower, give it its own shard rather than blending it with fast Chromium specs.
# quick per-shard timing from the merged blob set, to spot the long pole
npx playwright merge-reports --reporter=json ./all-blobs \
| jq '.suites[].specs[] | {title, shard: .tests[0].annotations, ms: .tests[0].results[0].duration}'
If shard durations differ by more than roughly 30 percent, rebalance — bumping matrix.shard count only helps once the load is even. Beyond balance, trimming the suite’s absolute cost (fewer redundant captures, cheaper fixtures) is the subject of reducing CI runner minutes for large map visual suites.
Verification
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| One shard runs far longer than the others | Uneven load — the tile-heavy, high-zoom captures all hashed into the same shard | Move heavy specs across the shard key boundary or split that engine into its own shard (step 6); equal test counts do not mean equal cost |
| The same baseline appears diffed on two shards | The shard key cut through a page’s states, so two shards loaded the same style and captured the shared baseline | Group whole, disjoint units per shard key (step 1) so no baseline belongs to more than one shard |
| A shard passes but the merged check is missing specs | A runner hit the out-of-memory killer (exit 137) and its blob report was partial or never uploaded | Lower per-shard concurrency (workers), add a shard to thin each one’s memory, and gate the merge on !cancelled() so partial results surface instead of vanishing |
Frequently asked questions
Should I shard by test file or by --shard=i/n?
Use --shard=i/n for the split itself — it deterministically hashes the whole test list into n even buckets, so you do not hand-maintain a file-to-shard mapping. Reserve file or project grouping (step 1) for the shard key: it decides which specs are eligible to land together, so a shard owns whole, self-contained units. The two work in tandem — grouping keeps shards self-contained, --shard distributes the groups.
Why merge blob reports instead of just reading each shard's result?
Because the gate must be one verdict, not n. Playwright HTML reports cannot be combined, so each shard emits a machine-mergeable blob; merge-reports stitches them into a single HTML report and re-derives one aggregate exit code. That exit code becomes the single required status check on the pull request, keeping the gate exactly as strict as the single-runner version — any one shard’s diff fails the whole check.
Do more shards always make the suite faster?
Only until fixed per-job overhead dominates. Each shard pays for runner startup, npm ci, browser image pull, and fixture restore before it captures a single frame. Past a point, adding shards buys less capture parallelism than it costs in setup, and an uneven split means the slowest shard — not the count — bounds wall-clock. Balance load first (step 6), then add shards only while the long pole keeps shrinking.
How do I stop shards from rendering pixels differently from each other?
Present byte-identical conditions on every runner: pin the browser through one container image, force the swiftshader software GL backend, fix the device scale factor, and serve tiles from content-addressed fixtures rather than a live CDN (step 4). If a spec’s diff changes depending on which shard runs it, one shard’s environment has drifted — usually a stale fixture cache or an unpinned GL path.