Calibrating Map Diff Thresholds from CI Telemetry

Most map suites set their first thresholds by trying numbers until the suite goes green, and then never revisit them. That is defensible on day one and indefensible six months later, because by then the environment has changed several times and nobody can say whether the numbers still describe the noise or merely the last incident. The measurements needed to do better are already being produced on every green run; this procedure turns them into calibrated values with a recorded provenance, and into a monthly proposal that tightens them as the suite improves.

This is a task within Visual Gate Threshold Configuration in CI, under CI/CD & Visual Testing Operations. The values it produces are stored in the profile described in Writing a Map Diff Threshold Profile as Code.

Prerequisites

Step-by-step procedure

1. Record the measurement on every run, including the passing ones

The instinct is to record failures. The useful data is the opposite: a green run’s measured diff is the noise floor, sampled under real conditions. Append one record per capture per region class, with the engine, the runner class and the image digest attached.

Where the telemetry that calibrates a threshold comes from Every green run publishes one record per capture carrying the measured changed-pixel ratio and SSIM for each region class. Those records accumulate into a telemetry store, from which a rolling window is taken. Percentiles are computed over the window per region class and per engine, and the ninety-ninth percentile is what a proposed tolerance is derived from. An arrow returns from the proposal to a pull request against the profile, so a human still approves the number. A caption notes that failing runs are deliberately excluded, because a failure is a measurement of something other than noise. Green runs are the only honest sample of the noise floor green run one record per capture telemetry store append-only p99 per class, per engine rolling window proposed value as a pull request failing runs excluded — a failure measures something else a human still approves the number; the pipeline only proposes it

2. Take the window wide enough to include the unlucky runs

Two weeks or a hundred runs, whichever is longer, and spanning every runner class in use. A window taken from one runner class measures that class; a window taken over three days measures three days. The tail is what a tolerance has to survive, and the tail is exactly what a short window misses.

3. Use the p99, not the maximum and not the mean

The mean describes a typical run and fails one run in two. The maximum is a single observation and is dominated by whatever the worst outlier happened to be — often a run that was already anomalous for an unrelated reason. The ninety-ninth percentile describes the realistic worst case while remaining a statistic rather than an anecdote.

4. Choose the margin from the spread, not from habit

A safety margin chosen from the spread rather than from habit A distribution of measured diff scores from green runs is drawn as a curve with a long right tail. Three candidate tolerances are marked. At the ninety-fifth percentile the tolerance sits inside the distribution, so one run in twenty fails for noise. At the ninety-ninth percentile plus a small margin the tolerance sits just past the observed tail, which is the recommended placement. Far to the right, a tolerance chosen as a round number sits well beyond anything ever measured, leaving a wide band in which a genuine regression would pass unnoticed. A caption notes that the right-hand gap is the suite blind spot, and that it is invisible unless the distribution is plotted. The gap to the right of the tail is the blind spot you chose p95 1 in 20 fails p99 × 1.3 recommended a round number a regression landing in this band passes silently

A multiplier of about 1.3 on the p99 is a reasonable default for a stable environment. What matters more than the exact number is that the resulting tolerance sits just past the observed tail: everything beyond that point is a band in which a genuine regression passes unnoticed, and it is invisible unless someone plots the distribution.

5. Write the value, the percentile and the date together 

regionClasses:
  labels:
    changedPixelRatio: 0.0025
    note: "p99 0.0019 over 132 runs, 2026-07-14 to 2026-07-28, chromium-121 + swiftshader"

The note is what makes the value defensible when it blocks a release. Without it, the only available argument in the moment is that the number is inconvenient, and that argument always wins.

6. Re-measure after any environment change

Which measurements have to be re-taken after each kind of change Four kinds of change are listed against what they invalidate. A browser or GL backend change invalidates every class, because anti-aliasing moves across the whole frame. A font package change invalidates the label class and nothing else. A style change invalidates whichever classes the edited layers belong to. A runner class change invalidates nothing directly but widens the spread, so the window has to be re-taken rather than the values recomputed from the old one. A caption states the rule: re-measure the affected classes rather than nudging the existing numbers until the suite goes green. Re-measure what changed; never nudge until it passes browser or GL backend every class — full re-measure font package labels only style edit the classes the edited layers belong to runner class nothing directly, but the spread widens a value nudged during an incident is indistinguishable from a calibrated one six months later, and that is what the measured-percentile note prevents

7. Run the monthly tightening proposal

Compare each class’s configured tolerance against its current p99 and open a pull request wherever the configured value exceeds it by more than the margin. Attach the distribution to the pull request so the reviewer is approving evidence rather than a number. Over a year this is what turns tolerances set defensively during rollout into tolerances that reflect what the suite actually does — typically three to five times tighter, with every step traceable.

8. Watch the ratio between the noise floor and the faults you catch

The calibration above answers “how loose must the gate be”. A second number answers the question that actually matters: how much room is left between the noise and a real fault.

Take the p99 of green-run scores for a class — the noise floor — and the median score of confirmed regressions in that class from the review history. The ratio between them is the gate’s separation. A separation of ten or more means the two populations barely overlap and the threshold’s exact placement hardly matters. A separation under about three means the gate is discriminating between distributions that are nearly touching, and every failure is a coin flip dressed as a measurement.

A low separation is not fixed by tuning. It is fixed upstream, by removing noise: pinning something that was not pinned, masking something volatile, tightening a readiness predicate that lets a frame through slightly early. That is why the separation is worth tracking as its own metric — it is the number that tells you whether more calibration work is worth doing, or whether the effort belongs in determinism instead.

Teams that plot it usually find one or two region classes with a poor separation and the rest comfortable, and the poor ones are almost always label regions on scenarios where something in the font or glyph path is not fully controlled. That is a much more actionable finding than “the labels class needs a looser tolerance”, which is the conclusion the same data produces if only the noise floor is looked at.

Verification

Confirm the procedure worked before wiring it into a blocking gate:

Troubleshooting

Symptom Likely cause Fix
Proposed tolerances oscillate month to month The window is too short, so each month’s p99 is dominated by different outliers Widen to at least a hundred runs, and require a proposal to move a value by more than a threshold before it is opened
The p99 is far above the mean for one class only That class contains a region whose noise is bimodal — commonly a label region on a scenario where a font falls back intermittently Investigate the bimodality rather than raising the tolerance; a second mode is usually a determinism bug with a specific cause
Percentiles differ sharply between two runner classes The fleet is not homogeneous, and the shared tolerance is calibrated for the noisier class Either pin the suite to one runner class or scope the profile by runner, the same way it is scoped by engine

Frequently asked questions

Why not just use the maximum observed value?

Because a maximum is one observation, and the run that produced it is disproportionately likely to have been anomalous for a reason unrelated to rendering noise — a runner under exceptional load, a transient network condition that the fixture layer had not yet covered. Calibrating to it bakes that anomaly into the gate permanently. The p99 keeps the realistic worst case while remaining a property of the distribution.

Should failing runs contribute to the telemetry at all?

Not to the calibration, because a failing run measures whatever broke rather than the noise floor. It is worth recording them separately, though: the distribution of failing scores is what tells you whether the gate is catching things comfortably outside the noise or scraping along just past the threshold, and the latter is a sign the tolerance and the fault sizes are too close together to be reliable.

How does this interact with a suite that is still being built?

Set the initial values defensively, mark them as unmeasured in the note, and let the first month of telemetry replace them. What matters is that the unmeasured state is visible in the profile rather than implicit, so the audit has something to prompt on. A suite whose initial guesses were never revisited looks identical to one that was calibrated, which is precisely the problem.

Can the same telemetry drive the SSIM floor?

Yes, with the direction reversed: the floor is derived from a low percentile rather than a high one, because SSIM decreases as difference increases. Take the first percentile of green-run scores per class and place the floor slightly below it. The reasoning about window width, margins and re-measurement is unchanged.