Dynamic Threshold Configuration

A single global pixel-diff tolerance is the wrong tool for slippy maps. Map rendering engines composite vector geometry, raster basemaps, anti-aliased labels, and GPU-shaded terrain in the same frame, and each of those layers tolerates a different amount of benign variation. Pin the tolerance too tight and harmless anti-aliasing noise fails every CI run; loosen it enough to silence that noise and a genuine regression in a dense urban core slips through. Dynamic threshold configuration resolves the conflict by computing the comparison tolerance per region from the rendering context — the zoom level, the layer mix, the device pixel ratio, and the GPU backend — instead of applying one number everywhere.

This page extends the capture and diff internals documented in Screenshot Capture, Sync & Comparison Logic; it assumes you already have deterministic capture in place and focuses purely on how the diff stage decides what counts as a failure.

What a Dynamic Threshold Actually Is #

A dynamic threshold is a function, not a constant. Where a static comparator asks “does the per-pixel difference exceed 0.1?”, a dynamic comparator evaluates a tolerance that varies across the image as a function of measurable inputs. The effective tolerance for a region can be modelled as:

$${\tau }_{\text{eff}}(z,d)={\tau }_0\cdot m_{\text{layer}}\cdot \left(1+\alpha {\log }_2\frac{z}{z_0}\right)\cdot (1+\beta d)$$

where ${\tau }_0$ is the base tolerance, $m_{\text{layer}}$ is a per-layer multiplier (vector geometry < raster tiles < label glyphs), $z$ is the current zoom relative to a reference zoom $z_0$, $d$ is local label/feature density, and $\alpha ,\beta$ are tuning coefficients calibrated against historical false-positive rates. The comparator then flags a region only when its measured delta exceeds ${\tau }_{\text{eff}}$ for that region.

Two distance metrics feed this decision. Raw per-pixel RGBA difference catches gross structural regressions but over-reacts to sub-pixel shifts; the Structural Similarity Index (SSIM) measures perceptual similarity and is far more tolerant of anti-aliasing jitter:

$$\mathrm{SSIM}(x,y)=\frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+c_2)}$$

where $\mu$, ${\sigma }^2$, and ${\sigma }_{xy}$ are the local means, variances, and covariance of baseline patch $x$ and candidate patch $y$, and $c_1,c_2$ stabilise the ratio over low-variance regions such as solid ocean fills. A robust threshold engine combines both: a pixel-count gate to catch large structural breaks and an SSIM gate to suppress perceptual noise. The trade-offs between these metrics are explored in depth in Diff Algorithm Tuning for Cartography.

Architecture: Profiles, Telemetry, and Region Maps #

A maintainable threshold system has three structural pieces, kept separate so each can be versioned and reviewed independently.

Tolerance profiles. Externalise tolerance matrices into version-controlled YAML rather than hard-coding them in test files. A profile maps an environment fingerprint — HEADLESS_CHROMIUM, MOBILE_WEBKIT, PRODUCTION_GPU — to a set of base tolerances and per-layer multipliers. CI runners select the active profile from an environment variable, so the same test code enforces appropriately different allowances on a SwiftShader CPU runner versus a hardware-GPU runner.

Capture telemetry. The comparator cannot compute ${\tau }_{\text{eff}}$ without knowing the conditions under which a frame was rendered. Every baseline must be stored with a sidecar metadata record: geographic bounds, exact zoom, device pixel ratio, the WEBGL_debug_renderer_info string, and the timestamp at which the render reached quiescence. This is the same metadata contract that Baseline Management for Tile Servers prescribes for versioned baselines, and reusing it keeps a single source of truth.

Region maps. Tolerance is spatial. A region map assigns each part of the viewport to a tolerance class — strict for vector geometry and raster alignment, relaxed for text and drop shadows, and fully excluded for transient UI. Region maps can be expressed as GeoJSON exclusion polygons (resolved through the map’s projection back into screen space) or as DOM-selector masks for chrome such as attribution badges and scale bars. Coordinating these exclusions with the broader masking strategy is the job of the Dynamic Element Masking & UI Stability work; threshold configuration consumes those masks rather than redefining them.

The original tiered routing — strict, relaxed, masked — still drives the per-region decision:

Step-by-Step Implementation #

The following procedure wires a dynamic threshold engine into a Playwright-based pipeline. It assumes capture synchronization is already solved upstream.

1. Pin the rendering context before capture. Dynamic thresholds are only meaningful when every variable except the change under test is fixed. Normalise the camera state with non-animated calls and round fractional zoom to a fixed precision so tile coordinates do not drift between runs — the discipline detailed in Viewport & Zoom Sync Strategies.

   await page.evaluate(() => {
     const map = window.__MAP__;
     map.jumpTo({
       center: [-73.9857, 40.7484],
       zoom: Math.round(14.37 * 100) / 100,
       bearing: 0,
       pitch: 0,
     });
   });
   

2. Wait for render quiescence, then snapshot telemetry. Only compute a diff once the tile pipeline has drained, as covered in Handling Async Tile Loading. Capture the metadata in the same step so the baseline and the candidate are described identically.

   const meta = await page.evaluate(() => new Promise((resolve) => {
     const map = window.__MAP__;
     map.once("idle", () => {
       const gl = map.painter.context.gl;
       const dbg = gl.getExtension("WEBGL_debug_renderer_info");
       resolve({
         bounds: map.getBounds().toArray(),
         zoom: map.getZoom(),
         dpr: window.devicePixelRatio,
         renderer: dbg ? gl.getParameter(dbg.UNMASKED_RENDERER_WEBGL) : "unknown",
       });
     });
   }));
   

3. Resolve the active tolerance profile. Load the YAML profile keyed by the runner environment, falling back to a conservative default so an unrecognised runner fails closed rather than silently passing.

   profiles:
     HEADLESS_CHROMIUM:
       base_tolerance: 0.004      # 0.4% per-pixel delta budget
       ssim_floor: 0.985
       layer_multipliers:
         vector: 1.0
         raster: 1.6
         labels: 2.5
     MOBILE_WEBKIT:
       base_tolerance: 0.007
       ssim_floor: 0.978
       layer_multipliers:
         vector: 1.2
         raster: 1.8
         labels: 3.0
   

4. Compute the effective per-region tolerance. Apply the zoom and density terms of ${\tau }_{\text{eff}}$ to each region class using the captured telemetry, then hand the region map plus tolerances to the comparator.

   function effectiveTolerance(profile, layer, zoom, density, z0 = 12) {
     const alpha = 0.08, beta = 0.5;
     const m = profile.layer_multipliers[layer] ?? 1.0;
     const zoomTerm = 1 + alpha * Math.log2(zoom / z0);
     return profile.base_tolerance * m * zoomTerm * (1 + beta * density);
   }
   

5. Run the dual-gate comparison. Evaluate both the pixel-count gate and the SSIM floor. A region fails only when it breaches the gate appropriate to its class, which prevents anti-aliasing jitter in label regions from tripping the strict vector gate.

   const result = compare(baseline, candidate, {
     regions: regionMap,
     tolerance: (region) =>
       effectiveTolerance(profile, region.layer, meta.zoom, region.density),
     ssimFloor: profile.ssim_floor,
   });
   if (!result.pass) throw new Error(`Regression in ${result.failingRegions.join(", ")}`);
   

6. Emit drift telemetry, never auto-loosen. Record the measured delta per region to a time series. If a region’s delta trends upward across runs without a code change, raise a baseline-drift alert for human review instead of widening the tolerance automatically.

Cross-Browser and Cross-Environment Considerations #

Threshold profiles exist because rendering is not portable. The divergences that most often force per-environment tolerances are:

• GPU floating-point variance. Chromium on a hardware GPU, WebKit on Apple silicon, and a SwiftShader CPU fallback produce different rasterisations of the same gradient fill or extruded 3D feature, frequently differing by one to two pixels along edges. Either pin a single rendering backend across all runners (the most deterministic choice) or maintain a distinct profile per backend.
• Device pixel ratio. A DPR of 2 quadruples the pixel count and changes how anti-aliasing distributes alpha across glyph edges. Tolerances calibrated at DPR 1 will misbehave at DPR 2; bake DPR into the profile fingerprint.
• OS font stacks and hinting. ClearType, GDI, and CoreText hint glyphs differently, so label regions need the highest layer multipliers and benefit most from the SSIM gate. Preloading the map style’s web fonts before initialisation removes glyph-substitution flicker that no tolerance can safely absorb.
• WebGL context attributes. Force consistent context flags — antialias: false, preserveDrawingBuffer: true — so the framebuffer you read back matches what was composited. Residual WebGL artifacts that survive this hardening are better handled upstream in Noise Reduction for Map Artifacts than by inflating thresholds.

Run the threshold engine inside the same container image across local and CI environments so font packages, GPU drivers, and headless flags cannot drift between where a baseline was authored and where it is enforced.

Threshold & Parameter Reference #

The following values are reasonable starting points for MapLibre GL / Mapbox GL workloads; calibrate against your own false-positive history before tightening them.

Common Pitfalls #

• Blanket tolerance inflation. Raising one global tolerance to silence label noise also blinds the vector gate to real geometry regressions. Fix: keep tolerances per-region and per-layer; never tune the global number to fix a localised problem.
• Capturing before quiescence. A threshold tuned against a half-rendered baseline encodes the wrong noise floor and oscillates between pass and fail. Fix: gate every comparison on a verified idle signal and a minimum stabilisation window before computing ${\tau }_{\text{eff}}$.
• Silent auto-promotion of baselines. Pipelines that auto-update the baseline whenever a diff “looks small” launder regressions into the reference set. Fix: require manual sign-off or multi-run statistical agreement before promoting, and alert on upward drift rather than absorbing it.
• Profile fingerprint omits DPR or GPU. Reusing one profile across DPR 1 and DPR 2 runners, or across hardware and software GL, produces tolerances that are simultaneously too tight on one runner and too loose on another. Fix: include DPR and the unmasked renderer string in the profile key.
• Masking and thresholds fighting each other. Defining exclusion zones twice — once in the masking layer and again as a near-infinite tolerance — leaves stale rules that drift apart. Fix: let the masking system own exclusions and have the threshold engine consume them as region classes.

Frequently Asked Questions #

Should I use pixel difference or SSIM for map thresholds?

Use both as independent gates. The per-pixel count catches large structural breaks such as a missing layer or shifted tile grid, while the SSIM floor tolerates the sub-pixel anti-aliasing jitter that dominates label regions. A region passes only when it clears the gate assigned to its layer class.

How do I keep thresholds from drifting looser over time?

Treat tolerance widening as a code change that needs review, and store per-region deltas as a time series. When a region trends upward without a corresponding source change, raise a baseline-drift alert for a human rather than letting an automated job relax the number. Auto-promotion should require manual sign-off or agreement across several runs.

Why does the same threshold pass locally but fail in CI?

Almost always a context mismatch: a different GPU backend, device pixel ratio, font package, or headless flag between your machine and the runner. Pin the rendering context in a shared container image and include DPR and the GPU renderer string in the profile fingerprint so each environment enforces its own calibrated tolerance.

What zoom and density inputs should feed the tolerance function?

The exact zoom captured at idle and a measure of local feature or label density per region. Higher zooms and denser label clusters legitimately produce more anti-aliasing variation, so the $\alpha$ (zoom) and $\beta$ (density) terms widen tolerance precisely where benign noise concentrates, without loosening sparse regions.

Related #

• Up to the parent reference: Screenshot Capture, Sync & Comparison Logic
• Deep dive on this topic: Configuring pixel diff thresholds for anti-aliased map labels
• Sibling: Viewport & Zoom Sync Strategies
• Sibling: Handling Async Tile Loading
• Sibling: Noise Reduction for Map Artifacts
• Cross-topic: Diff Algorithm Tuning for Cartography