Scaling & Cost of Map Visual Test Suites

A map visual regression suite that was cheap at fifty screenshots becomes a line item at five thousand. Each captured view is a headless browser session that boots an engine, hydrates a tile grid, waits for the GPU to settle, and writes a PNG — seconds of billed runner time that then multiply across every rendering engine, every device pixel ratio, and every zoom level you promised to protect. Add the storage of baselines and per-run diff artifacts, and the human minutes spent triaging failures, and the suite develops three cost curves that grow at different rates and for different reasons. This page builds an explicit model of those curves, shows how the cross-engine baseline matrix multiplies the capture count, and works through the levers — change-based selection, caching, retention policies, ephemeral runners, and DPR trade-offs — that bend the total back down without quietly deleting coverage.

This area sits under CI/CD Integration & Visual Test Operations, the parent guide for running map visual tests as a gating pipeline. It assumes you already have a working gate — the mechanics of wiring one live in GitHub Actions & GitLab CI Gates for Map Visual Tests — and focuses only on what that gate costs at scale and how to keep the bill proportional to the risk it retires.

What “cost” actually decomposes into #

The instinct is to treat suite cost as a single number — the CI invoice — but that hides the three independent drivers you can actually tune. A map visual suite spends money in three places, each governed by a different variable.

• Compute is billed runner time: every second a shard spends booting the browser, pulling baselines, launching the map, waiting for tiles, capturing, and diffing. It scales with the total number of captures and the mean seconds per capture, and it is the dominant cost for most suites because map captures are slow — a hydrated WebGL frame is far heavier than a static DOM snapshot.
• Storage is the standing cost of the baseline set plus the churning cost of diff and candidate artifacts each run produces. Baselines are a fixed inventory that grows with the matrix; diffs are a flow that grows with run frequency and failure rate.
• Review is engineering time: the minutes a QA engineer or cartographer spends deciding whether a flagged diff is a regression or acceptable drift. It scales with the failing capture count, which is dominated by flakiness far more than by real defects — the economics of which are the subject of Flaky Visual Test Triage & Quarantine.

Writing this as one expression makes the leverage points explicit. For a single CI run over a suite of $N$ captures spread across $S$ parallel shards, with mean capture-plus-diff time $\bar{t}$ seconds, a per-shard fixed boot cost of $b$ seconds, and a runner rate $\rho$ in dollars per minute:

$$C_{\text{run}}\approx \underset{\text{compute}}{\underbrace{\frac{\rho }{60}\left(Sb+N\bar{t}\right)}}+\underset{\text{baselines + diffs}}{\underbrace{C_{\text{storage}}}}+\underset{\text{review}}{\underbrace{rNmw}}$$

The review term expands the intuition above: $r$ is the fraction of captures that fail and reach a human, $m$ is the mean minutes to adjudicate one, and $w$ is the loaded wage per minute. The single most important structural fact in that formula is that sharding appears twice and pulls in opposite directions. Wall-clock time falls with more shards:

$$T_{\text{wall}}\approx b+\frac{N\bar{t}}{S}$$

but the billed compute term $Sb+N\bar{t}$ rises with $S$, because every shard pays its own boot cost $b$ — the browser install, the baseline pull, the map warm-up. Sharding buys latency with money. Past a point, adding shards barely moves wall-clock (the $N\bar{t}/S$ term is already small relative to $b$) while each new shard keeps adding its fixed $b$ to the invoice. The optimum is not “maximum parallelism”; it is the knee of that curve.

The matrix multiplies the capture count #

The reason map suites cross the cost threshold so fast is that $N$ is not a list of screenshots — it is a product. Every distinct thing you actually want to protect is a view: a place, at a zoom, at a viewport, in a UI state. But you cannot compare a view across engines with a single baseline, because Chromium, WebKit, and Firefox rasterize the same MapLibre frame differently — the divergence catalogued in Cross-Browser Baseline Matrix. Each engine needs its own golden. So does each device pixel ratio, because DPR-2 doubles the tile grid and changes anti-aliasing. The capture count is therefore multiplicative:

$$N=V\times E\times D\times Z$$

where $V$ is distinct views, $E$ is rendering engines, $D$ is DPR variants, and $Z$ is any additional axis you fan out (themes, locales, breakpoints). A modest-looking suite of 200 views, on 3 engines, at 2 DPRs, with a light and dark theme, is not 200 captures — it is $200\times 3\times 2\times 2=2400$. At two seconds of billed capture-and-diff each, that is 80 runner-minutes of pure capture per run before any boot overhead, plus 2400 baselines to store and re-bless on every style change.

This is why the matrix is the first thing to interrogate when cost climbs. Every axis you add multiplies not just compute but the standing baseline inventory and the re-blessing labour each style release triggers. The discipline that keeps the baseline inventory itself manageable — deduplicated, versioned, keyed to a style hash rather than copied per branch — is covered in Baseline Management for Tile Servers; a bloated or duplicated baseline store makes both the storage term and the review term worse at the same time.

Design patterns that break the multiplication #

You cut a multiplicative cost by attacking the factors, not by shaving seconds off $\bar{t}$. Four structural patterns do most of the work, and they compose.

Change-based selection removes views from $V$ on any given run. Most commits do not touch most of the map: a change to the water color affects every view, but a change to one overlay’s label font affects only the handful of views that render that overlay at a zoom where its labels are visible. If you can compute, from the diff, which style layers and which tile fixtures a commit touches, you can capture only the views whose rendered output could have changed and skip the rest — trusting the last green baseline for the untouched majority. On a typical pull request this turns a 2400-capture run into a few hundred, and it is the highest-leverage lever because it scales the entire $N\bar{t}$ term down per run without weakening the nightly full-matrix pass.

Snapshot and dependency caching removes the fixed boot cost $b$ from as many shards as possible. The browser binary, the font packages, node_modules, and — crucially for maps — the tile and style fixtures are identical run to run; caching them turns a 90-second cold shard into a 15-second warm one. Because $b$ is paid per shard and rises with $S$, shrinking $b$ is what makes higher parallelism affordable when you genuinely need the wall-clock.

Tile-fixture reuse removes redundant hydration work and network variance at once. If every view that covers downtown San Francisco at zoom 12 pulls from one shared, version-pinned fixture set rather than a live tile server, you pay the fixture storage once and every capture that uses it hydrates deterministically and fast. This is the same determinism that makes the gate trustworthy, now doing double duty as a cost control.

Retention and compression policies bound the storage flow. Diff artifacts and candidate PNGs are write-once, read-during-triage, delete-after; keeping them for 90 days at full resolution is pure waste. Compress diff artifacts, keep them only for open failures plus a short window, and never let a per-run flow accumulate into the baseline inventory.

The economics of choosing self-hosted runners and open-source stacks over a managed platform — where these levers land differently because the vendor prices compute and storage together — are worked through in the cost analysis of cloud visual testing for mapping apps.

Step-by-step: building a cost-aware run #

The following procedure wires the levers into a single run so the model above becomes a control panel rather than a spreadsheet. It assumes a working gate and a version-pinned fixture set.

1. Instrument the three cost terms. Before optimizing, measure. Emit per-run counters for captures executed, mean capture-plus-diff seconds, shard boot seconds, diff bytes written, and failures sent to review. Without these you cannot tell which term dominates, and the dominant term is the only one worth touching.

   // Emit a machine-readable cost record at the end of every run.
   const record = {
     captures: results.length,
     meanCaptureSec: mean(results.map((r) => r.durationMs / 1000)),
     shardBootSec: process.env.SHARD_BOOT_SEC,
     diffBytes: results.reduce((n, r) => n + (r.diffBytes ?? 0), 0),
     reviewCount: results.filter((r) => r.status === 'needs-review').length,
   };
   fs.writeFileSync('cost-record.json', JSON.stringify(record, null, 2));
   

2. Compute the changed-view set. Diff the current commit against the merge base, map changed paths to affected style layers and fixtures, and resolve the set of views whose output could differ. Fall back to the full matrix when the change touches shared infrastructure (the base style, the capture harness, the engine version).

   const changed = execSync('git diff --name-only origin/main...HEAD')
     .toString().trim().split('\n');
   
   const touchesEverything = changed.some((p) =>
     /^(styles\/base|harness\/|package-lock\.json)/.test(p));
   
   const views = touchesEverything
     ? allViews
     : allViews.filter((v) =>
         v.layers.some((l) => changed.includes(`styles/layers/${l}.json`)) ||
         v.fixtures.some((f) => changed.includes(`fixtures/${f}`)));
   

3. Choose the shard count from the model, not from habit. Pick $S$ near the knee where added shards stop cutting wall-clock. A quick heuristic: shard until each shard runs for roughly the boot time $b$, i.e. $S\approx \sqrt{N\bar{t}/b}$, which is where marginal wall-clock savings and marginal boot cost balance.

   const N = views.length * engines.length * dprs.length;
   const meanCaptureSec = 2.0, bootSec = 15;
   const shards = Math.max(1, Math.round(Math.sqrt((N * meanCaptureSec) / bootSec)));
   console.log(`Sharding ${N} captures across ${shards} shards`);
   

4. Warm every shard from cache. Restore the browser binary, fonts, node_modules, and the tile/style fixtures before the first capture so $b$ is dominated by restore, not install. In GitHub Actions this is a keyed actions/cache; the fixture cache key is the fixture manifest hash, so a fixture change invalidates exactly the affected shards.

   - uses: actions/cache@v4
     with:
       path: |
         ~/.cache/ms-playwright
         fixtures/tiles
       key: vr-${{ hashFiles('fixtures/manifest.json', 'package-lock.json') }}
   

5. Capture at the cheapest DPR that still catches the defect class. Run the full matrix at DPR-1 and reserve DPR-2 for the small subset of views where retina anti-aliasing is itself the thing under test. DPR-2 doubles the tile grid, roughly doubles hydration time, and quadruples the pixel area a diff must scan.

   const dprs = view.retinaSensitive ? [1, 2] : [1];
   

6. Compress and scope diff retention. Write diff artifacts as compressed PNGs, upload them only on failure, and set a short retention so the storage flow cannot accumulate. Baselines live in versioned object storage; diffs do not belong there.

   - uses: actions/upload-artifact@v4
     if: failure()
     with:
       name: visual-diffs
       path: diff/
       compression-level: 9
       retention-days: 7
   

The full, runnable treatment of these steps against a large real suite — including spot-runner fallback and per-view budgets — is expanded in reducing CI runner-minutes for large map visual suites.

Cross-environment considerations #

Cost optimizations interact with the correctness controls in the rest of this area, and a saving that reintroduces flake is not a saving — it just moves the cost from compute to review.

• Spot and ephemeral runners are the largest compute discount available, often 60–90% off on-demand rates, and map captures are a near-ideal fit because a shard is short-lived and idempotent: if a spot instance is reclaimed mid-shard, retry that shard’s slice. The one caveat is that a fresh ephemeral runner pays full boot cost $b$, so spot savings and aggressive caching must be designed together.
• DPR-1 versus DPR-2 is a coverage decision, not just a cost one. DPR-1 catches structural regressions — a missing layer, a broken fill, a dropped label — at a quarter of the pixel-scan area. DPR-2 additionally catches retina-specific anti-aliasing and hairline-stroke defects. Running the whole matrix at DPR-2 “to be safe” quadruples diff area and doubles hydration for a defect class most views never exhibit; reserve it per view.
• Engine coverage tiers. Not every view needs every engine on every commit. A common split is: all views on Chromium per pull request (the cheap, deterministic baseline engine), and the full cross-engine matrix nightly. This keeps $E$ at 1 on the hot path and 3 on the scheduled pass — the maintenance strategy for those per-engine goldens is detailed in Cross-Browser Baseline Matrix.
• Change-based selection must fail safe. A selection bug that skips a view which did change is a missed regression — a false negative, the worst outcome for a gate. Always fall back to the full matrix when a change touches shared code, and run the complete suite on a schedule so nothing can hide behind selection indefinitely.
• Storage tiering. Baselines are read on every run and belong on standard object storage; diff artifacts are read only during triage and can go to cheaper, short-retention tiers. Do not pay hot-storage prices for write-once diff PNGs.

Cost levers reference #

Each lever trades spend for some risk. Start at the top — change-based selection and caching are close to free wins — and adopt lower rows only with the stated guardrail in place.

Common pitfalls #

Adding shards stopped making CI faster but the bill kept climbing

Root cause: you pushed $S$ past the knee, where $N\bar{t}/S$ is already small next to the per-shard boot cost $b$, so each new shard adds a full $b$ to billed compute while barely moving wall-clock. Diagnose: plot wall-clock and total runner-minutes against shard count over recent runs; if wall-clock has flattened while minutes rise linearly, you are past the knee. Fix: set $S\approx \sqrt{N\bar{t}/b}$, and attack $b$ with caching before adding shards — a lower boot cost moves the knee rightward and makes parallelism cheaper.

Change-based selection let a real regression through

Root cause: the map from changed files to affected views was incomplete — a shared base-style token, an engine bump, or a harness change altered output for views the selector considered untouched. Diagnose: the nightly full-matrix run flags a regression that the PR run skipped; compare the two view sets. Fix: treat shared infrastructure paths as “touches everything” and fall back to the full matrix, and always keep a scheduled full run so selection can only ever defer coverage, never delete it.

Storage costs are growing faster than the suite

Root cause: diff and candidate artifacts are being retained like baselines — uncompressed, long-retention, on hot storage — so a per-run flow accumulates into a standing inventory. Diagnose: break the storage bill into baseline bytes versus diff bytes; a diff share that grows with run frequency rather than matrix size is the tell. Fix: compress diffs, upload only on failure, set short retention, and keep diffs off the versioned baseline store entirely. Keep baselines deduplicated and keyed to a style hash per Baseline Management for Tile Servers.

Review time dominates the real cost even though compute looks fine

Root cause: the $r$ term — fraction of captures reaching a human — is inflated by flake, not by defects, so engineers spend their minutes re-adjudicating the same non-deterministic diffs. Diagnose: measure how many reviewed diffs are dismissed as “no real change”; a high dismissal rate is flake tax, not review load. Fix: cut $r$ at the source with determinism and quarantine per Flaky Visual Test Triage & Quarantine before optimizing compute, because review minutes are the most expensive unit in the model.

Spot-runner savings evaporated into retries and boot time

Root cause: shards were not idempotent, so a reclaimed spot instance forced a full-suite rerun rather than re-running its slice, and every fresh instance paid an uncached boot cost. Diagnose: compare spot reclaim frequency against rerun cost; if a single reclaim reruns far more than its shard’s work, the shard is not isolated. Fix: make each shard capture a disjoint, resumable slice keyed by shard index, retry only the reclaimed slice, and pair spot runners with a warm cache so a replacement instance boots in seconds.

Frequently asked questions #

Does parallelizing across more shards reduce my CI bill?

No — it reduces wall-clock, not billed compute. Total runner-minutes are roughly $Sb+N\bar{t}$, which rises with shard count $S$ because every shard pays its own fixed boot cost $b$. Sharding buys latency with money. To cut the bill itself, reduce the capture count $N$ with change-based selection or shrink $b$ with caching; add shards only when wall-clock, not cost, is the constraint.

What is the single highest-leverage way to cut a large map suite's cost?

Change-based test selection, because map suite cost is dominated by the multiplicative capture count $N=V\times E\times D\times Z$ and most commits touch only a few views. Capturing only the views whose style layers or tile fixtures a commit changed turns a full-matrix run into a small fraction of it per pull request, while a scheduled nightly full run guarantees nothing hides behind selection.

Should I run every view at DPR-2 to be safe?

No. DPR-2 roughly doubles hydration time and quadruples the pixel area each diff scans, for a defect class — retina-specific anti-aliasing and hairline strokes — that most views never exhibit. Run the full matrix at DPR-1 to catch structural regressions cheaply, and reserve DPR-2 for the specific views where retina rendering is itself under test.

Related #

• CI/CD Integration & Visual Test Operations — the parent guide to running map visual tests as a gating pipeline.
• Reducing CI runner-minutes for large map visual suites — the fully wired procedure behind the compute lever.
• Cross-Browser Baseline Matrix — how the engine axis multiplies capture count and how to tier it.
• GitHub Actions & GitLab CI Gates for Map Visual Tests — wiring the gate whose cost this page models.
• Cost analysis of cloud visual testing for mapping apps — managed versus self-hosted economics for the same levers.
• Baseline Management for Tile Servers — keeping the baseline inventory deduplicated and versioned so storage and review stay bounded.

← Back to CI/CD Integration & Visual Test Operations