Reducing CI Runner Minutes for Large Map Visual Suites

A map visual suite grows quadratically without anyone deciding it should: every new basemap style, every zoom level, every device pixel ratio, and every viewport multiplies into another screenshot, and each screenshot pays for tile fetches, a WebGL warm-up, and a GPU-composited capture. A suite of a thousand map views on a cold runner can burn well over an hour of billed minutes per push, and most of that time is spent re-doing work that did not change. This guide lists the concrete levers that cut that bill — what each one saves, how to wire it, and how to confirm it did not weaken the gate — with a worked before/after example you can drop your own numbers into.

This is one task inside Scaling & Cost of Map Visual Test Suites, the guide that treats runner spend as a first-class engineering constraint, and it sits under the broader CI/CD Integration & Visual Test Operations area that covers how map visual gates run in a pipeline. It assumes you already have a working suite and a green gate; the goal here is to make that same gate cheaper, not to change what it asserts.

Prerequisites

A working map screenshot suite (Playwright or Puppeteer over MapLibre GL, Mapbox GL, Leaflet, or OpenLayers) that already passes deterministically.
A CI provider that bills per runner-minute and supports a build cache (GitHub Actions, GitLab CI, or similar).
A per-run timing breakdown — setup, capture, diff, upload — so you can see which stage dominates before you optimize it.
Tile, style, and glyph assets you can serve as local fixtures instead of fetching from a live CDN.
A test-to-source mapping (or the ability to derive one) linking each map view to the style, component, and fixture files it depends on.

A cost model to target

Before touching anything, write the spend down as a formula so each lever has a term it moves. Billed minutes for a sharded run are the per-shard setup replicated across every shard plus the capture work, which is divided among shards but summed back when you pay per shard-minute:

M = setup p \cdot o + capture V \cdot \frac{t}{60}

where $p$ is the shard count, $o$ the per-shard setup minutes (browser install, dependency install, server boot), $V$ the number of views captured, and $t$ the seconds per view. Every lever below drives one of those four variables down. The two that dominate a large suite are $V$ (attack it with change-based selection) and $o$ (attack it with caching), and shard tuning keeps $p \cdot o$ from quietly eating the savings.

Step-by-step procedure

1. Select only the views whose inputs changed

The largest single saving is not capturing views that could not have changed. Build a dependency map from each map view to the files it renders from — its style JSON, the components mounted in that view, and its tile and glyph fixtures — then intersect the pushed diff against that map and run only the affected views. A typical pull request touches one style or one component and legitimately needs a fraction of the suite.

// scripts/select-views.mjs — emit the views a diff can actually affect
import { execSync } from 'node:child_process';
import { readFileSync } from 'node:fs';

// view -> [globs] it renders from
const deps = JSON.parse(readFileSync(new URL('./view-deps.json', import.meta.url)));

const changed = execSync('git diff --name-only origin/main...HEAD')
  .toString().trim().split('\n').filter(Boolean);

const match = (globs) => globs.some((g) =>
  changed.some((f) => new RegExp(g.replace(/\*/g, '.*')).test(f)));

const selected = Object.entries(deps)
  .filter(([, globs]) => match(globs))
  .map(([view]) => view);

// Shared inputs (base style, map wrapper) force a full run — fail safe, not open.
const universal = /src\/map\/core|styles\/base\.json|playwright\.config/;
process.stdout.write(
  changed.some((f) => universal.test(f)) ? 'ALL' : selected.join('\n'));

The rule that keeps this honest: when a shared input changes — the base style, the map wrapper, the runner config — fall back to the full suite. Selection must fail toward running more, never less, or it becomes a way to miss regressions.

2. Cache the browser image and node_modules

A cold runner reinstalls the browser binary and every dependency on every run — often the single biggest slice of $o$ . Cache both, keyed on the lockfile and the pinned browser version so the cache invalidates exactly when it must and never when it must not.

# .github/workflows/visual.yml (excerpt)
- uses: actions/setup-node@v4
  with: { node-version: 20, cache: npm }

- name: Cache Playwright browsers
  uses: actions/cache@v4
  with:
    path: ~/.cache/ms-playwright
    key: pw-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

- name: Install deps (offline where cached)
  run: npm ci --prefer-offline --no-audit --no-fund

- name: Install browser only on cache miss
  run: npx playwright install --with-deps chromium
  if: steps.pw-cache.outputs.cache-hit != 'true'

A warm browser-and-deps cache commonly takes per-shard setup from a couple of minutes to well under a minute, and because setup is multiplied by every shard, that saving is amplified by $p$ .

3. Serve tile and glyph fixtures instead of the network

Fetching live tiles over the network makes every capture pay round-trip latency and reintroduces the flake that pinned fixtures exist to remove. Record the tiles, sprites, glyphs, and style JSON once, commit them behind a manifest keyed to a style version, and serve them from localhost. This is the same asset-versioning discipline described in Baseline Management for Tile Servers, applied here for speed rather than correctness.

// Serve committed fixtures; fail loudly on an un-recorded request so the
// suite can never silently reach the live CDN and re-introduce latency.
await context.route(/\.(pbf|mvt|png|webp|json)(\?|$)/, async (route) => {
  const key = route.request().url().replace(/^https?:\/\/[^/]+/, '');
  const file = path.join('fixtures', key);
  if (fs.existsSync(file)) return route.fulfill({ path: file });
  return route.abort('failed'); // surfaces a missing fixture instead of a slow fetch
});

Local fixtures cut $t$ directly — a capture that spent one to two seconds waiting on tiles now spends none — and remove the network variance that would otherwise force retries.

4. Tune the shard count to the sweet spot

Sharding is not free: each shard re-pays $o$ , so the $p \cdot o$ term grows linearly with shard count while the capture term shrinks. Past a point, adding shards raises total billed minutes to save wall-clock time you do not need. Find the count where marginal shards stop paying for themselves.

# Sweep shard counts against the selected view count and per-shard setup,
# print billed minutes so you can pick the knee of the curve.
V=180; t=1.6; o=0.6   # views, seconds/view, setup minutes per shard
for p in 2 4 6 8 12 20; do
  minutes=$(echo "$p * $o + $V * $t / 60" | bc -l)
  printf 'shards=%2d  billed=%.1f min\n' "$p" "$minutes"
done

For a selected run the sweet spot is usually small — enough shards to fit the PR deadline, not the maximum the plan allows. Distributing the selected views evenly is the subject of parallelizing map screenshot tests across CI shards; the point here is that on a change-based run you almost always want fewer shards than on a full run.

5. Drop DPR-2 to DPR-1 where it is safe

A device-pixel-ratio-2 (retina) capture quadruples pixel area, which inflates both capture time and diff time, and doubles tile demand. Reserve DPR-2 for the handful of views where hairline strokes or label anti-aliasing genuinely regress at DPR-1, and render everything else at DPR-1.

// playwright.config.js — DPR-1 is the default; a small tagged set opts into DPR-2.
export default defineConfig({
  projects: [
    { name: 'maps-dpr1', use: { deviceScaleFactor: 1 } },
    { name: 'maps-dpr2', use: { deviceScaleFactor: 2 },
      grep: /@retina/ }, // only views tagged @retina pay for 2x
  ],
});

Keeping DPR-1 and DPR-2 baselines separate is a rendering requirement regardless — mixing them in one baseline is a documented source of false diffs — so this lever costs nothing in coverage when applied deliberately.

6. Compress and expire diff artifacts

A failing map diff writes three PNGs — baseline, actual, and the highlighted difference — per view. On a broad regression that is thousands of images, and uploading and retaining them at full size adds upload minutes and storage cost. Compress before upload, upload only on failure, and set a short retention window.

- name: Upload diffs (failures only, compressed, short-lived)
  if: failure()
  uses: actions/upload-artifact@v4
  with:
    name: map-diffs
    path: test-results/**/*-diff.png
    compression-level: 9
    retention-days: 5

Skipping artifact upload on green runs — the overwhelming majority — removes the upload stage entirely from the common path, and a five-day retention keeps storage from accruing indefinitely for images no one revisits.

7. Run the suite on ephemeral spot runners

Capture is embarrassingly parallel and fully restartable, which is exactly the workload spot and ephemeral runners are priced for. Run the map suite on a spot pool at a fraction of on-demand cost, and let CI reschedule any shard the provider reclaims — a re-run of one interrupted shard is cheap because selection already shrank the work.

runs-on:
  group: spot-linux          # preemptible pool, per-minute rate well below on-demand
  labels: [self-hosted, gpu-swiftshader]
strategy:
  fail-fast: false           # a reclaimed shard should not cancel the others

This lever does not change $M$ in the formula above — it lowers the price of each billed minute rather than the count — but it compounds with every minute the earlier steps already removed.

The numbers plug straight into the model. Before: $M = 20 \cdot 2.5 + 1200 \cdot \frac{4}{60} = 50 + 80 = 130$ minutes. After change-based selection ( $V = 180$ ), warm caches ( $o = 0.6$ ), fixtures and DPR-1 ( $t = 1.6$ ), and shard tuning ( $p = 6$ ): $M^{'} = 6 \cdot 0.6 + 180 \cdot \frac{1.6}{60} = 3.6 + 4.8 = 8.4$ minutes — a 94% reduction before spot pricing lowers the per-minute rate on top. Your suite’s constants will differ; the shape of the win will not.

Verification

Confirm each lever saved minutes without weakening the gate:

Selection is fail-safe. Push a change to a shared style or the map wrapper and confirm the run expands to the full suite, not a subset.
A known regression still fails. Introduce a deliberate one-pixel style change to a selected view and confirm the gate goes red — savings that hide regressions are not savings.
Cache hit rate is high. Check the run log shows browser and dependency cache hits on an unrelated PR; a persistent miss means the cache key is churning.
No live network in capture. Confirm the fixture route aborts un-recorded requests and the run makes zero external tile calls.
Billed minutes dropped. Compare the provider’s billed-minutes report for a representative PR before and after; the drop should track the model, not just wall-clock time.

Troubleshooting

Symptom	Likely cause	Fix
Billed minutes barely moved after adding shards	The $p \cdot o$ setup term grew as fast as the capture term shrank	Re-run the shard sweep in step 4 and pick the knee; on selected runs use fewer shards
A regression slipped through a green run	Change-based selection skipped the affected view because its dependency map was incomplete	Widen the view’s globs, and route shared inputs to a full run per step 1
Cache never hits, setup stays cold	Cache key includes a value that changes every run, or the browser version is unpinned	Key the cache on the lockfile hash plus a pinned browser version so it invalidates only on real change

Frequently asked questions

Won't change-based selection let regressions through?

Only if the dependency map is wrong or the fallback is missing. Build the map from each view to the style, component, and fixture files it renders, and route any change to a shared input — base style, map wrapper, runner config — to the full suite. Selection must fail toward running more views, never fewer, so a green selective run is trustworthy. Verify it by pushing a shared-input change and confirming the run expands.

Which lever should I apply first?

Measure first, then attack the largest term. On most large suites the capture term $V \cdot t /60$ dominates, so change-based selection (cutting $V$ ) and tile fixtures plus DPR-1 (cutting $t$ ) pay back fastest. If a per-run breakdown shows setup dominating instead — many shards, cold installs — start with caching and shard tuning. The cost analysis of cloud visual testing for mapping apps walks through pricing the stages so you know which term is largest.

Is dropping DPR-2 to DPR-1 a loss of coverage?

Not if you keep DPR-2 for the views that need it. Hairline strokes, thin borders, and dense label anti-aliasing can regress at high density in ways DPR-1 misses, so tag those views to run at DPR-2 and render the rest at DPR-1. Because DPR-1 and DPR-2 baselines must be stored separately regardless, splitting them by need costs nothing in coverage and removes the 4x pixel penalty from the bulk of the suite.

Do spot runners make the suite flakier?

They add one failure mode — a reclaimed shard — which is not a test flake and is cheap to absorb. Disable fail-fast so one reclaimed shard does not cancel the others, and let CI reschedule the interrupted shard. Because change-based selection already shrank the work, re-running a single shard is a small cost, and the per-minute price of spot capacity is well below on-demand.

Up to the parent guide Scaling & Cost of Map Visual Test Suites, and the grandparent area CI/CD Integration & Visual Test Operations.
Parallelizing map screenshot tests across CI shards — distributing the selected views once you have tuned the shard count.
Cost analysis of cloud visual testing for mapping apps — pricing the stages so you know which term to attack first.
Baseline Management for Tile Servers — versioning the tile and glyph fixtures this guide serves from local disk.

← Back to CI/CD Integration & Visual Test Operations

Reducing CI Runner Minutes for Large Map Visual Suites

Prerequisites #

A cost model to target #

Step-by-step procedure #

1. Select only the views whose inputs changed #

2. Cache the browser image and node_modules #

3. Serve tile and glyph fixtures instead of the network #

4. Tune the shard count to the sweet spot #

5. Drop DPR-2 to DPR-1 where it is safe #

6. Compress and expire diff artifacts #

7. Run the suite on ephemeral spot runners #

Verification #

Troubleshooting #

Frequently asked questions #

Related #