Per-Region Tolerance Classes for Map Interfaces

A single tolerance for a whole map frame has to be set for the noisiest thing in it, which on any cartographic render is the label glyphs. That number then applies to the water fill occupying most of the frame — a region whose rasteriser has no legitimate reason to produce a different pixel at all. The result is a gate that tolerates in flat colour exactly the change it needs to tolerate in text, which is several times more than a real fill regression would produce. Region classes fix that, and the part worth getting right is that the map of classes is generated rather than drawn.

This is a task within Visual Gate Threshold Configuration in CI, under CI/CD & Visual Testing Operations. The classes it produces are the ones the profile in Writing a Map Diff Threshold Profile as Code assigns tolerances to.

Prerequisites

Step-by-step procedure

1. Generate the class map from the style and the frame

Where a region class comes from, and why it is generated rather than drawn A region map is produced from two inputs rather than by hand. The style supplies which layers exist and what type each is, so fills, lines, symbols and raster layers are already labelled. The rendered frame supplies where each layer actually painted, queried from the map at capture time. Combining them yields a per-pixel class map with no manual step. A crossed-out alternative shows the hand-drawn approach: rectangles someone traced once, which are correct for one viewport, one zoom and one style version, and silently wrong after any of the three changes. Generate the region map; a traced rectangle is correct once style: layer types fill, line, symbol, raster frame: where each painted queried at capture time per-pixel class map hand-traced rectangles right for one viewport, once a generated map follows a viewport change, a zoom change and a style edit for free a traced one is silently wrong after any of the three, and nothing reports it

At capture time, ask the map which layers are rendered and what type each is, then render a class map by drawing each layer’s own geometry into an offscreen buffer with its class as the colour. For raster and hillshade layers the extent is the whole viewport; for vector layers it is whatever the layer painted.

const classes = await page.evaluate(() => {
  const map = window.__testMap;
  return map.getStyle().layers
    .filter((l) => map.getLayer(l.id) && map.getLayoutProperty(l.id, 'visibility') !== 'none')
    .map((l) => ({ id: l.id, type: l.type }));
});

2. Resolve overlapping layers by noise precedence

Overlapping layers and the precedence that resolves them A single pixel can belong to several layers at once — a label drawn over a road drawn over a landuse fill. The classification therefore needs a precedence, and the correct order is by how much the topmost layer can legitimately vary. Symbol pixels win, because glyph hinting is the noisiest thing in the frame and a label edge over a road should be judged by label rules. Line pixels come next for join and cap rasterisation. Raster pixels follow. Fill pixels come last, because they are the quietest and any pixel that is only a fill should be held to the tightest tolerance. A caption warns that the opposite order lets a strict fill rule fail on a label edge that merely moved a subpixel. Noisiest layer wins the pixel 1 · symbol glyph hinting is the noisiest thing in the frame 2 · line join and cap rasterisation 3 · raster resample differences 4 · fill quietest — tightest tolerance reverse this and a strict fill rule fails on a label edge that moved a subpixel

3. Exclude masked regions from every class

A masked rectangle belongs to no class. Folding it into whichever class it overlaps quietly inflates that class’s measured noise, which then inflates its calibrated tolerance — a mask making the gate looser everywhere is not an outcome anyone intends.

4. Score each class separately 

const perClass = {};
for (const [name, mask] of Object.entries(classMasks)) {
  perClass[name] = compare(baseline, candidate, {
    include: mask,
    threshold: profile.regionClasses[name].changedPixelRatio,
  });
}

5. Take the verdict from the worst class

Scoring per class and taking the verdict on the worst one A frame is split by class into four scores, each compared against its own tolerance. The fill class scores well inside its tight budget. The line class is comfortable. The raster class is comfortable. The symbol class is over its budget despite that budget being the loosest, which is what a real label regression looks like. The verdict is taken from the worst class rather than from an average, so the capture fails. A caption notes that averaging the four would have produced a comfortable overall score and passed the capture, which is precisely the failure per-class scoring exists to prevent. Verdict from the worst class, never from the average fill line raster symbol over budget — fails averaging these four produces a comfortable score and passes the capture

6. Publish the per-class breakdown, not just the verdict

The breakdown is what makes a failure diagnosable at a glance: a capture failing on symbol alone is a label problem, one failing on fill alone is a style or colour problem, and one failing on every class at once is an environment problem. That triage happens before anyone opens an image.

7. Keep the class list short

Four or five classes cover a cartographic frame. Teams that grow to fifteen — a class per layer group — end up with classes that have too few pixels to calibrate, percentiles computed over a handful of measurements, and a profile nobody can read. If a class cannot accumulate enough telemetry to produce a stable percentile, it is not a class.

8. Decide what a class means before naming one

The four classes above are named after layer types, which is convenient and slightly misleading. What a class actually represents is a population of pixels whose noise has one cause, and keeping that definition in mind is what stops the list growing.

Symbol pixels form a class because glyph rasterisation is one mechanism producing one distribution of variation. Fill pixels form a class because a solid colour has essentially no legitimate variation, which is a different distribution with a different shape. Line pixels form a class because join and cap geometry produces variation at edges but not in the interior. Raster pixels form a class because resampling produces variation everywhere at low amplitude.

Layer type happens to correlate with those mechanisms almost perfectly, which is why naming classes after types works. It stops working when a team starts adding classes for reasons that are not about noise mechanisms — a class for “the layers the data team owns”, a class for “layers added since March”. Those are useful groupings for other purposes and terrible tolerance classes, because the pixels inside them have no common noise behaviour and their percentile describes a mixture rather than a distribution.

The practical test before adding a class is one sentence: what physical mechanism makes these pixels vary, and is it different from the mechanisms already covered? If the answer names an ownership boundary or a release date rather than a rasteriser, the grouping belongs somewhere other than the threshold profile.

Verification

Confirm the procedure worked before wiring it into a blocking gate:

Troubleshooting

Symptom Likely cause Fix
Every capture fails on the fill class The class map is assigning label or line pixels to fill, usually because precedence runs the wrong way Apply the precedence in step 2 — noisiest layer wins the pixel
One class has wildly unstable percentiles It covers too few pixels to produce a stable measurement Merge it into a neighbouring class; a class that cannot be calibrated is not doing any work
A masked region shows up in a class’s measured noise Masks are applied at capture but not excluded from the class map Subtract the mask from every class before scoring, as in step 3

Frequently asked questions

Does per-class scoring slow the comparison down?

Marginally. The comparison already visits every pixel; classifying it is a lookup into a buffer the same size as the frame. The measurable cost is generating the class map, which requires querying the style and rendering the layer extents once per capture — tens of milliseconds. Against the cost of a false failure or a missed regression, it does not register.

What about interface chrome that is not part of the map?

Give it its own class. Panels, legends and controls are DOM-rendered rather than rasterised by the map, so their noise profile is different again — usually much quieter than anything on the canvas, because no GL is involved. A chrome class with a very tight tolerance catches layout regressions that a canvas-calibrated number would absorb entirely.

Can classes differ per scenario?

The classes should not, but their tolerances may. A night-mode scenario has the same four classes as a day-mode one; what differs is that its raster class contains a hillshade with more dithering. That is a tolerance question, answered by a scenario override with a recorded reason, rather than a reason to invent a fifth class that exists only on one scenario.

How does this relate to the diff algorithm?

It is orthogonal and complementary. Region classes decide which pixels a given tolerance applies to; the algorithm choice — changed-pixel count, largest cluster, SSIM — decides how the difference in those pixels is measured. The combination described in Diff Algorithm Tuning for Cartography runs all three measures per class.