Tuning SSIM Thresholds for Vector Basemap Diffs

A structural similarity gate is only as good as the two numbers you feed it: the window size it scores over, and the floor below which a frame fails. Set the floor too high and every hinting shift on a re-rendered label halo trips the build; set it too low and a whole missing arterial road slides through as “similar enough.” Vector basemaps make this worse than ordinary UI diffing, because a legitimate re-render moves thousands of edge pixels by a subpixel while the map is structurally identical. This page walks through calibrating a windowed SSIM comparator on a labelled corpus of good and bad diffs so anti-aliasing and font hinting stay under the floor while a dropped road, a vanished label, or a broken polygon fill lands above it — with a floor picked per zoom band rather than one global guess.

This is one task within Diff Algorithm Tuning for Cartography, the guide that weighs structural against perceptual comparison for layered maps, and it sits under the broader discipline described in Web Map Visual Testing Fundamentals & Toolchains. It assumes your captures are already deterministic; if two runs of the same commit disagree, calibrate nothing until that is fixed.

Prerequisites

A stabilized capture pipeline that produces byte-consistent frames for an unchanged commit — the SSIM floor is meaningless over flaky input.
A structural-similarity library (ssim.js, image-ssim, or OpenCV’s SSIM) callable from Node, plus pngjs for decoding.
A labelled corpus: 30–60 diff pairs per zoom band, each tagged good (a legitimate re-render — anti-aliasing, hinting, driver dither) or bad (a real fault — missing road, dropped label, broken fill).
Frames grouped by zoom band (0–4, 5–9, 10–13, 14–18) so you can pick a floor per band rather than one global value.
A basemap-only baseline separated from overlay layers, so overlay churn cannot mask a basemap regression — see step 6.

Step-by-step procedure

1. Capture a stable good/good pair as the noise floor

Before scoring failures, measure what “no change” actually costs in SSIM. Capture the same locked viewport twice from the same commit and score them against each other. The result is your empirical ceiling on benign noise: whatever SSIM two identical-intent frames produce is the drift you must tolerate. Pin an integer zoom, disable easing, and force the software GL backend so the only differences are rasterizer noise.

import { PNG } from "pngjs";
import fs from "node:fs";
import ssim from "ssim.js";

const a = PNG.sync.read(fs.readFileSync("stable/home-z12-run1.png"));
const b = PNG.sync.read(fs.readFileSync("stable/home-z12-run2.png"));

// mssim is the mean SSIM over all windows; expect ~0.995+ on a stable pair.
const { mssim } = ssim(a, b, { windowSize: 11, bitDepth: 8 });
console.log(`benign-noise SSIM floor: ${mssim.toFixed(4)}`);

If this pair scores below roughly 0.99, your capture is not deterministic enough to calibrate against — the noise floor and the fault signal will overlap and no threshold can separate them.

2. Compute windowed SSIM with a deliberate window size

SSIM never scores the whole image at once. It slides an $N \times N$ window across both frames and computes a local score from luminance, contrast, and structure, then averages the per-window scores into the mean SSIM (MSSIM). For two aligned windows $x$ and $y$ :

SSIM (x, y) = \frac{( 2 μ _{x} μ _{y} + c _{1} ) ( 2 σ _{x y} + c _{2} )}{( μ _{x}^{2} + μ _{y}^{2} + c _{1} ) ( σ _{x}^{2} + σ _{y}^{2} + c _{2} )}

where $μ$ is the window mean, $σ^{2}$ the variance, $σ_{x y}$ the covariance, and $c_{1} = (k_{1} L)^{2}$ , $c_{2} = (k_{2} L)^{2}$ stabilize the ratio for a dynamic range $L$ (255 for 8-bit) with $k_{1} = 0.01$ , $k_{2} = 0.03$ . The window size is the load-bearing knob for maps. A small window (5–7) reacts to single-pixel edge shifts and over-reports anti-aliasing as structural loss; a large window (13–15) smooths over a thin missing road until it disappears into the average. An 11×11 Gaussian window is the canonical default and the right starting point for a basemap, because a road casing or label stroke spans several pixels and survives that smoothing while sub-pixel AA noise averages out.

// Score one pair across candidate window sizes to see how the choice moves MSSIM.
for (const windowSize of [7, 9, 11, 13]) {
  const { mssim } = ssim(baseline, current, { windowSize, bitDepth: 8 });
  console.log(`window ${windowSize}: MSSIM ${mssim.toFixed(4)}`);
}

3. Sweep the floor against the labelled corpus

With the window fixed, score every pair in the corpus and find the floor that best separates good from bad. Sweep candidate floors from 0.95 to 0.999 and, at each, count how many good pairs you would wrongly fail (false positives) and how many bad pairs you would wrongly pass (false negatives). The best floor sits in the gap between the two score distributions — high enough to catch the worst benign noise, low enough to still fail the mildest real fault.

const scored = corpus.map((p) => ({
  label: p.label, // "good" | "bad"
  mssim: ssim(PNG.sync.read(fs.readFileSync(p.baseline)),
              PNG.sync.read(fs.readFileSync(p.current)),
              { windowSize: 11, bitDepth: 8 }).mssim,
}));

for (let floor = 0.95; floor <= 0.999; floor += 0.001) {
  const falsePos = scored.filter((s) => s.label === "good" && s.mssim < floor).length;
  const falseNeg = scored.filter((s) => s.label === "bad" && s.mssim >= floor).length;
  console.log(`${floor.toFixed(3)}  FP=${falsePos}  FN=${falseNeg}`);
}

Read the sweep like an ROC trade-off: as the floor rises, false negatives fall and false positives climb. For a merge-blocking gate, prefer the floor that drives false negatives to zero first, then take the lowest such floor to keep false positives minimal. A missing road that ships is far more expensive than a re-run.

4. Pick the floor per zoom band

One floor across all zooms is a compromise that fits none of them. Low zooms are generalized coastlines and gradients where a re-render moves large smooth regions and MSSIM stays high, so the floor can be looser. High zooms are dense with fine text and hairline road casing where the same fault removes far fewer pixels, so the floor must be tighter to keep sensitivity. Run the step-3 sweep independently for each band and record the winning floor and window per band.

{
  "ssim": {
    "window": 11,
    "bands": [
      { "zoom": "0-4",   "floor": 0.975, "window": 11 },
      { "zoom": "5-9",   "floor": 0.982, "window": 11 },
      { "zoom": "10-13", "floor": 0.988, "window": 11 },
      { "zoom": "14-18", "floor": 0.992, "window": 9  }
    ]
  }
}

These are starting values from a typical Noto-labelled basemap; re-derive them from your own corpus. The pattern — floor rising with zoom, window tightening at street level — is what to preserve. This mirrors the raster-diff budgets in per-zoom diff threshold tables for slippy maps; keep the two tables consistent so a frame is not judged loosely by one gate and tightly by the other.

5. Wire the per-band floor into the comparator

Load the band table, resolve the current frame’s zoom from its baseline manifest, and fail the run when MSSIM drops below that band’s floor. Emit the score and the applied floor on every run so a reviewer sees why a frame passed or failed, not just the verdict.

import { readFileSync } from "node:fs";

const bands = JSON.parse(
  readFileSync(new URL("./ssim.bands.json", import.meta.url))
);

function floorFor(zoom) {
  return bands.ssim.bands.find(({ zoom: z }) => {
    const [lo, hi] = z.split("-").map(Number);
    return zoom >= lo && zoom <= hi;
  });
}

export function assertSsim(baseline, current, zoom) {
  const band = floorFor(zoom);
  const { mssim } = ssim(baseline, current, { windowSize: band.window, bitDepth: 8 });
  const passed = mssim >= band.floor;
  console.log(`z${zoom} MSSIM ${mssim.toFixed(4)} vs floor ${band.floor} → ${passed ? "pass" : "FAIL"}`);
  if (!passed) throw new Error(`SSIM regression at z${zoom}: ${mssim.toFixed(4)} < ${band.floor}`);
}

Structural scoring complements rather than replaces per-pixel diffing; the trade-offs are worked through in comparing pixel diff vs structural diff for GIS overlays.

6. Guard against basemap-vs-overlay masking

A single MSSIM over the composited frame can hide a basemap regression behind a stable overlay, or fail the whole frame because a legitimately churning overlay moved. Score the basemap layer and the overlay layers separately: render the basemap alone against its immutable golden with the tight floor from step 4, and score overlays against relaxed floors or mask them out entirely. This is the layer-isolation discipline that noise reduction for map artifacts applies upstream — remove the churn before it reaches the comparator so a broken road never averages out against a busy marker layer.

// Two independent gates instead of one composited score.
assertSsim(basemapGolden, basemapCurrent, zoom);       // tight floor, immutable golden
assertSsim(overlayBaseline, overlayCurrent, zoom, {    // relaxed floor for churning data
  floorOverride: band.floor - 0.02,
});

Verification

Confirm the calibration holds before it gates real merges:

Separation check. Plot the corpus scores per band; the lowest good score must sit above the highest bad score with a visible gap. If the clusters overlap, tighten the window or improve capture determinism before trusting any floor.
Zero false negatives. No bad pair scores at or above its band floor. A single missing road passing the gate invalidates the floor.
Injected-fault run. Delete one road layer from a fixture and re-score; MSSIM must drop below the floor for that band. If it does not, the window is too large for that feature’s stroke width.
Benign re-render run. Re-render the basemap with a driver-level AA change only; MSSIM must stay above the floor. A failure here means the floor is too tight and will generate re-run noise.

A correct calibration shows a clean gap between the two clusters in every band, catches the injected fault, and lets the benign re-render through.

Troubleshooting

Symptom	Likely cause	Fix
Good and bad clusters overlap, no clean floor exists	Window too small, so anti-aliasing scores as structural loss; or the capture pair is not deterministic	Raise the window toward `11`, then re-verify the step-1 stable pair scores above `0.99` before sweeping again
A whole missing road passes the gate	Window too large for the band, averaging a hairline feature into surrounding pixels; or floor set below the bad cluster	Drop to a `9×9` window at high zoom and re-run the step-3 sweep to pull the floor up into the gap
Frame fails only when a live overlay is present	One composited MSSIM masks basemap and overlay together, so overlay churn sinks the score	Split into per-layer gates (step 6): tight floor on the basemap golden, relaxed floor or mask on the overlay

Frequently asked questions

What SSIM window size should I start with for a vector basemap?

Start with an 11×11 Gaussian window. It is large enough that road casing, label strokes, and polygon edges — which span several pixels — survive the local averaging, while sub-pixel anti-aliasing and font-hinting noise wash out. Drop to 9×9 only at street-level zoom (14–18) where hairline features need tighter scoring, and re-run the labelled sweep whenever you change it.

Why does one global SSIM floor fail on maps?

Because the same fault removes a different fraction of pixels at each zoom. Low-zoom frames are smooth coastlines and gradients where MSSIM stays high and the floor can be loose; high-zoom frames are dense text and hairline roads where a dropped feature barely moves the mean, so the floor must be tight to stay sensitive. Sweep and set a floor per zoom band instead.

How many labelled diffs do I need to calibrate the floor?

Aim for 30–60 pairs per zoom band, split between legitimate re-renders and real faults, and make the faults realistic — a single dropped label, one missing road, a broken fill, not a blanked frame. The floor is only as trustworthy as the gap between the good and bad clusters, and too few samples leaves that gap undefined and the floor guessed.

Can SSIM replace pixel diffing entirely for maps?

No — treat them as complementary. SSIM tolerates anti-aliasing and catches structural loss, but it can miss a small, high-contrast color shift that a per-pixel diff flags immediately. Run a tuned pixel diff for local color faults and SSIM for structural integrity, and keep their per-zoom budgets consistent so a frame is not judged loosely by one and tightly by the other.

Up to the parent guide Diff Algorithm Tuning for Cartography, and the broader reference Web Map Visual Testing Fundamentals & Toolchains.
Comparing pixel diff vs structural diff for GIS overlays
Per-zoom diff threshold tables for slippy maps
Noise reduction for map artifacts

Tuning SSIM Thresholds for Vector Basemap Diffs

Prerequisites #

Step-by-step procedure #

1. Capture a stable good/good pair as the noise floor #

2. Compute windowed SSIM with a deliberate window size #

3. Sweep the floor against the labelled corpus #

4. Pick the floor per zoom band #

5. Wire the per-band floor into the comparator #

6. Guard against basemap-vs-overlay masking #

Verification #

Troubleshooting #

Frequently asked questions #

Related #