Cross-Browser Baseline Matrix

The moment a map visual suite runs on more than one browser engine, a single shared baseline stops being correct. Chromium, WebKit, and Firefox do not paint the same map from the same code: they carry different WebGL implementations, different font rasterizers, different subpixel anti-aliasing, and different glyph hinting. Feed all three the identical MapLibre GL style, the identical camera, and the identical tiles, and you still get three visibly distinct PNGs — off by a halo pixel here, a kerned label there, a gradient band somewhere else. Approve one of those frames as the baseline and the other two engines fail forever against a target they were never going to reproduce. The fix is not a looser threshold; it is a baseline matrix — a structured set of per-engine golden images, each keyed to the exact environment that produced it, so every engine is compared only against a frame that engine could plausibly draw. This page defines that matrix, its storage and naming model, which axes to include versus collapse, and how to keep it affordable as it expands in CI.

This page extends the operations layer of CI/CD Integration & Visual Test Operations, and it assumes you already gate a single-engine suite. Here the concern is the fan-out: what changes when the same scenarios must hold across three rendering engines at once, and how to stop that fan-out from multiplying your storage, runner minutes, and review load beyond what the added coverage is worth.

What a cross-browser baseline matrix is #

A baseline matrix is a versioned collection of golden images indexed by the axes that actually change the pixels, with exactly one golden per cell of that index. Where a single-engine suite stores one baseline per scenario, a matrix stores one baseline per (scenario × environment) pair, and treats “environment” as a small, explicit tuple rather than an accident of whatever machine last ran the test.

The load-bearing insight is that map rendering is a function of more inputs than the code under test. The visible frame depends on the browser engine, the engine version, the device pixel ratio, the map library and its version, and the resolved style. Two of those — engine and DPR — change anti-aliasing and glyph rasterization at the pixel level even when nothing about your application changed. So the matrix’s index key is:

(engine, engine-version, dpr, map-lib-version, style-hash)

Each unique tuple owns its own baseline set. A run computes its own tuple at capture time, looks up the matching golden, and diffs against that one — never against a golden captured under a different tuple. This is the same storage-separated-from-execution discipline described in setting up baseline image versioning for web maps, extended with the engine axis so a Chromium render is never held to a WebKit golden.

Why can’t one shared baseline work with a generous tolerance? Because the divergences are not uniform noise you can dial out — they are structured. WebKit’s CoreText hints glyphs onto a different subpixel grid than Chromium’s FreeType, so every label edge shifts consistently in one direction; that is a large, correlated pixel delta, not scattered anti-aliasing dither. Raise the threshold high enough to absorb it and you have simultaneously blinded the gate to a genuinely dropped label or a broken road casing, which produce deltas of the same magnitude. The whole value of the gate is discriminating real cartographic faults from rendering variance, and a shared baseline destroys exactly that discrimination. Per-engine goldens keep the tolerance tight because each engine is compared to a frame drawn by its own rasterizer. The engine-by-engine breakdown of where those divergences come from is worked through in Chromium vs WebKit vs Firefox MapLibre rendering divergence.

The matrix model: axes, keys, and cell ownership #

Think of the matrix as a grid whose axes are the tuple fields and whose cells each own a directory of golden images. The diagram below shows the common two-visible-axis projection — rendering engine against map library — with the remaining axes (engine-version, DPR, style-hash) folded into each cell’s key. Every cell owns its own baseline set, and a run only ever reads and writes the cell matching its captured tuple.

The key thing the grid makes visible is cell ownership. The Chromium/MapLibre cell owns eight goldens (one per sampled zoom), and it is the only thing a Chromium/MapLibre run ever compares against. Re-blessing that cell after an intentional style change touches nine images at most; it does not disturb the WebKit or Firefox cells, which still hold their own correct goldens for the unchanged engines. This isolation is what makes a large matrix maintainable — a change is contained to the cells it actually affects.

The full baseline count for the matrix is the product of the axis cardinalities and the per-cell scenario set:

$$N_{\text{baselines}}=\underset{\text{engines}}{\underbrace{E}}\times \underset{\text{versions}}{\underbrace{V}}\times \underset{\text{DPRs}}{\underbrace{D}}\times \underset{\text{libs}}{\underbrace{L}}\times \underset{\text{styles}}{\underbrace{S}}\times \underset{\text{scenarios}}{\underbrace{C}}$$

This product is why axis selection is the single most important design decision. Add DPR-2 across the board and you double $N$; add a fourth engine and you grow it by a third. The next section is about keeping every factor in that product justified.

Which axes to include versus collapse #

Not every axis earns a place in the key. An axis belongs in the matrix only if it changes the pixels in a way you need to catch a regression against. If two values of an axis produce visually identical frames, keying on it just doubles your storage and review for no coverage. Decide each axis deliberately.

• Engine — always include. This is the entire reason the matrix exists. Chromium, WebKit, and Firefox diverge structurally; each needs its own goldens. Do not collapse it.
• DPR — include only the ratios you ship and test. DPR changes tile count and anti-aliasing, so a DPR-1 golden is meaningless for a DPR-2 run. But most teams gate at DPR-1 in CI and treat DPR-2 as a smaller, targeted sweep rather than a full second grid. Include DPR-2 for the handful of scenarios where retina label crowding is a real risk; do not multiply the whole matrix by it.
• Engine-version — include, but pin coarsely. A minor Chromium bump rarely moves map pixels; a major one occasionally does (a new ANGLE backend, a FreeType update). Key on the major version, pin the exact browser build in your container image, and re-bless deliberately when you upgrade rather than tracking every patch release. Pinning is covered in Containerized Rendering Environments for Map Tests.
• Map-lib-version — include. MapLibre 4 and MapLibre 5, or Leaflet 1.9 versus a 2.x, can change default rendering. Different libraries entirely (MapLibre vs Leaflet vs OpenLayers) are effectively different products and always warrant separate cells — see maintaining per-engine baselines for Leaflet and OpenLayers.
• Style-hash — include as a cache key, not a multiplier. The style hash does not expand the matrix; it invalidates cells. When the resolved style changes, the affected cells’ goldens are stale and must be re-blessed. Storing the hash in the key means a run can detect “this golden was made for a different style” and fail loudly instead of diffing against something outdated.
• Zoom / scenario — collapse to a representative sample. You do not need a golden at every integer zoom. Sample the zoom ranges that render differently in character — a low-zoom generalized coastline, a mid-zoom road-and-label frame, a high-zoom street-level frame — as the per-zoom threshold reasoning in the parent toolchain already establishes.
• OS / GPU — collapse by removing the variance instead of keying on it. The host GPU is the axis you most want to eliminate rather than enumerate, which is the subject of the next section.

The rule of thumb: include an axis only if you cannot make its values converge, and collapse it the moment you can. Every axis you can collapse is a factor removed from $N_{\text{baselines}}$.

Software GL: collapsing the host-GPU axis #

The most dangerous hidden axis is the host GPU. If your renders depend on whatever graphics hardware happens to be in a runner, you have effectively added an uncontrolled dimension: the same Chromium build produces different WebGL output on an Intel iGPU, an NVIDIA card, and a cloud runner’s virtualized GPU, because each driver dithers gradients, blends translucency, and resolves multisampling slightly differently. That variance is not something to key on — you cannot maintain a golden per GPU model — it is something to remove.

The standard removal is a software GL backend. Launch Chromium with ANGLE targeting SwiftShader so WebGL is rasterized on the CPU with a deterministic reference implementation, identical on every runner regardless of host hardware:

chromium \
  --headless=new \
  --use-gl=angle \
  --use-angle=swiftshader \
  --disable-gpu \
  --force-color-profile=srgb \
  --force-device-scale-factor=1 \
  --font-render-hinting=none

With SwiftShader in place, the Chromium row of the matrix becomes host-independent: a developer’s laptop, a GitHub Actions runner, and a GitLab shared runner all produce the same Chromium/MapLibre frame. That is what lets a single Chromium golden serve every environment instead of forcing a per-runner sub-axis.

The catch is that not every engine offers the same escape hatch. Chromium’s ANGLE/SwiftShader path is mature; WebKit and Firefox under Playwright do not expose an equivalent software-GL flag, so their WebGL output is comparatively more sensitive to the host stack. The practical consequence is that you pin those engines’ environment even harder — same container image, same driver, same fonts — and lean on noise reduction for map artifacts to absorb the residual variance that no flag removes. The full container-pinning recipe lives in Containerized Rendering Environments for Map Tests.

Step-by-step: building and using the matrix in CI #

The following procedure stands up a matrix, keys captures to it, and expands it across engines in CI without letting cost run away. It assumes single-engine capture (camera locking, tile-load gating) is already solved upstream.

1. Define the axis tuple and a deterministic key function. Compute the key from values you can read at capture time, so a run always knows which cell it owns.

   import { createHash } from 'node:crypto';
   
   function baselineKey({ engine, engineVersion, dpr, mapLib, styleJson }, scenario, zoom) {
     const styleHash = createHash('sha256')
       .update(JSON.stringify(styleJson))
       .digest('hex')
       .slice(0, 8);
     const major = engineVersion.split('.')[0];
     return `${engine}/${major}/dpr${dpr}/${mapLib}/${styleHash}/${scenario}-z${zoom}.png`;
   }
   

2. Declare the matrix as data, not as duplicated test code. Keep the axes in one config so the fan-out is visible and tunable in a single place.

   export const matrix = {
     engines: ['chromium', 'webkit', 'firefox'],
     dprs: [1],                    // DPR-2 only for the retina subset, below
     mapLibs: ['maplibre-4.7'],
     scenarios: ['home', 'urban', 'terrain'],
     zooms: [4, 10, 16],
     retinaSubset: { scenarios: ['urban'], dprs: [2] },
   };
   

3. Capture into the cell keyed by the run’s own tuple. Read the real engine and version from the launched browser rather than assuming them, so a mismatched runner cannot write into the wrong cell.

   const version = browser.version();               // real build, not assumed
   const key = baselineKey(
     { engine, engineVersion: version, dpr, mapLib, styleJson },
     scenario, zoom,
   );
   const shot = await page.locator('#map').screenshot({ animations: 'disabled' });
   await store.put(key, shot);                       // object storage, keyed path
   

4. On comparison, resolve the exact cell — or fail loudly if it is missing. A missing golden for the current tuple is a signal (new engine, bumped style), not a licence to fall back to another engine’s golden.

   const golden = await store.get(key);
   if (!golden) {
     throw new Error(`No baseline for ${key} — bless this cell before gating`);
   }
   const changedRatio = diff(golden, shot, thresholdFor(zoom));
   

5. Expand the engines across CI shards, not inside one job. Run each engine as its own parallel job so a slow WebKit shard does not stall the Chromium gate, and so per-engine failures are independently visible. This is the sharding model detailed in GitHub Actions & GitLab CI Gates for Map Visual Tests.

   jobs:
     map-visual:
       strategy:
         fail-fast: false
         matrix:
           engine: [chromium, webkit, firefox]
       runs-on: ubuntu-22.04
       steps:
         - uses: actions/checkout@v4
         - run: npx playwright install --with-deps ${{ matrix.engine }}
         - run: npm run test:visual -- --engine=${{ matrix.engine }}
   

6. Re-bless per cell, never per grid. When an intentional change lands, update only the cells whose tuple actually changed. A style edit re-blesses every engine’s cell for that style hash; a Chromium upgrade re-blesses only the Chromium row.

   # Re-bless only the cells whose style hash changed, across all engines
   npm run baselines:bless -- --style-hash a4f9 --engines chromium,webkit,firefox
   # Re-bless only Chromium after a browser bump
   npm run baselines:bless -- --engine chromium --engine-version 122
   

Cross-browser and cross-environment considerations #

Standing the matrix up is the easy half; keeping each engine’s cells trustworthy is where the environment work concentrates.

• Chromium is the anchor: --use-gl=angle --use-angle=swiftshader makes its cells host-independent, so its goldens are the cheapest to keep stable and the safest to generate on a developer laptop for later CI comparison.
• WebKit under Playwright renders text through CoreGraphics-style rasterization with no software-GL flag, so its labels sit on a different subpixel grid than Chromium’s and its cells drift more with the container’s font stack. Pin fonts explicitly and never share a golden with Chromium.
• Firefox uses WebRender and a distinct glyph rasterizer; its vector-tile decode timing also differs, so its cells are most sensitive to under-synchronized capture. Harden tile-load gating before trusting a Firefox baseline.
• Font stack is the axis that silently corrupts cells across all engines. A missing font package substitutes a fallback face and shifts every label, invalidating goldens without any code change. Pin the exact font packages in the container image, the same discipline as Containerized Rendering Environments for Map Tests.
• DPR-2 retina doubles the tile count and changes anti-aliasing, so a DPR-2 run needs its own cell. Keep it a targeted subset — the scenarios where retina label crowding actually risks a regression — rather than a full second grid.

Per-engine divergence sources and mitigations #

Each row of the matrix diverges for concrete, identifiable reasons. Knowing the source tells you whether to remove it (pin/flag), absorb it (noise reduction), or accept it (per-engine golden).

Common pitfalls #

One shared baseline with a loose threshold "works" until it hides a real regression

Root cause: the threshold was raised high enough to absorb the structural WebKit-versus-Chromium label shift, which is a large correlated delta — the same magnitude as a genuinely dropped label. Diagnose: inject a real fault (delete a layer) and confirm the loose gate still passes; if it does, the tolerance is masking regressions. Fix: give each engine its own cell keyed by (engine, engine-version, dpr, map-lib-version, style-hash) and restore a tight per-cell threshold, since each engine is now compared to a frame its own rasterizer drew.

Chromium passes everywhere locally but the whole matrix flakes in CI

Root cause: renders depend on the host GPU, so a developer’s discrete GPU and the CI runner’s virtualized one dither differently — an uncontrolled axis. Diagnose: run the same commit on two different runners and diff the two current captures; a non-empty diff with no code change is host-GPU variance. Fix: launch Chromium with --use-gl=angle --use-angle=swiftshader --disable-gpu so WebGL rasterizes on the CPU identically everywhere, collapsing the host-GPU axis out of the matrix entirely.

A run diffs against a stale golden after a style change and passes when it should fail

Root cause: the baseline key did not include the style hash, so an edited style resolved against a golden captured under the old symbology. Diagnose: compare the run’s resolved style hash to the one recorded in the cell manifest; a mismatch means the golden is stale. Fix: put the style hash in the key so a changed style either resolves to a fresh cell or fails loudly as missing, forcing a deliberate re-bless of the affected cells across all engines.

The matrix cost triples the moment a third engine is added

Root cause: every axis was enumerated at full cardinality — three engines times two DPRs times every integer zoom — so $N_{\text{baselines}}$ exploded. Diagnose: compute the product of your axis cardinalities; if DPR-2 or per-integer-zoom is multiplying the whole grid, that is the cost. Fix: collapse what you can — sample representative zooms, make DPR-2 a targeted subset rather than a full second grid, pin engine versions coarsely — and shard engines across parallel CI jobs so wall-clock cost stays flat even as baseline count grows.

Re-blessing one engine accidentally overwrites another engine's goldens

Root cause: the bless step operated on a scenario path without the engine segment in the key, so it clobbered every engine’s cell for that scenario. Diagnose: check whether the golden paths include the engine and version segments; if a bless touches more files than the changed cells, the key is too coarse. Fix: key every golden path by the full tuple and scope the bless command to specific engines or style hashes, so a Chromium upgrade re-blesses only the Chromium row and a style edit re-blesses only the matching style-hash cells.

Frequently asked questions #

Can I share one baseline across engines if I mask the labels?

Only partially, and rarely worth it. Masking the text regions removes the glyph-hinting divergence, but WebGL backends still blend gradients and translucency differently, so basemap fills and overlay alpha continue to diverge across engines. You end up masking so much of the frame that the gate no longer covers the map. A per-engine golden keyed by (engine, engine-version, dpr, map-lib-version, style-hash) keeps full coverage at a tight threshold and is cheaper to reason about than an ever-growing mask.

Do I need a separate baseline for every browser patch version?

No. Key on the major engine version and pin the exact browser build in your container image. Minor and patch releases almost never move map pixels; a major bump occasionally does when it ships a new ANGLE backend or FreeType update. Re-bless the affected engine’s row deliberately when you upgrade, rather than tracking every patch — that keeps the version axis from multiplying the matrix.

When is a per-engine pixel difference acceptable versus a real regression?

A difference is an acceptable engine divergence when it is present between two correct runs of the same commit on that engine against its own golden — that is rendering variance, and it should already be absorbed by a tight per-cell threshold. It is a regression when the current render diverges from that engine’s own golden by more than the threshold, especially as a localized structural change (a missing road, a dropped label, a broken fill) rather than a uniform subpixel shift. The decision framework is: compare within a cell, never across cells; a cross-cell difference is expected and owned by the matrix, while an in-cell difference beyond tolerance is the fault the gate exists to catch.

Related #

• Up to the parent guide CI/CD Integration & Visual Test Operations — where the matrix sits in the wider operations pipeline.
• Chromium vs WebKit vs Firefox MapLibre rendering divergence — the engine-by-engine breakdown of exactly where the pixels differ.
• Maintaining per-engine baselines for Leaflet and OpenLayers — the matrix applied to raster-DOM and canvas map libraries rather than GL.
• Setting up baseline image versioning for web maps — the storage and manifest model the engine axis extends.
• Containerized Rendering Environments for Map Tests — pinning fonts and the GPU backend so each cell stays reproducible.
• Noise Reduction for Map Artifacts — absorbing the residual per-engine variance no flag removes.
• GitHub Actions & GitLab CI Gates for Map Visual Tests — sharding the engine axis across parallel CI jobs.

← Back to CI/CD Integration & Visual Test Operations