Writing a Map Diff Threshold Profile as Code

The tolerances a map suite enforces usually live in four places at once: a comparator default, an inline override in a spec, an environment variable added during an incident, and a job setting that softens the whole check. Nobody can answer what tolerance a given capture is actually running at without reading the pipeline, which means nobody reviews the numbers and they only ever get looser. This procedure moves them into one file, adds the schema rules that keep them honest, and makes every published result carry the provenance of the number it was judged against.

This is a task within Visual Gate Threshold Configuration in CI, under CI/CD & Visual Testing Operations. The numbers it records are derived by the method in Dynamic Threshold Configuration.

Prerequisites

Step-by-step procedure

1. Inventory every current tolerance without changing any of them

Grep the repository and the CI configuration for comparator options, thresholds, continue-on-error, and any if: that skips a visual job. Write each into the new profile at its current value with reason: "migrated, not yet reviewed". The suite behaves identically after this step, which is what makes it safe.

2. Give every entry the four fields that make it reviewable

Four fields that turn a tolerance into a reviewable decision A threshold entry is decomposed into four fields. The tolerance value is the number the comparator enforces, always expressed as a fraction rather than a percentage. The measured percentile records the ninety-ninth percentile the value was derived from, together with the date it was measured. The owner names who defends the value when it blocks a release. The review date is what the monthly audit reads to ask whether the value can be tightened. A caption notes that an entry missing any of the last three is indistinguishable from a guess. tolerance value the number the comparator enforces a fraction, never a percentage measured percentile the p99 it was derived from with the date it was measured owner who defends it when it blocks a release review date when the audit asks whether it can be tightened an entry missing any of the last three is indistinguishable from a guess

3. Define the region classes before the scenario overrides

Most scenario-level overrides turn out to be the same problem written down several times. Defining classes first means each of those becomes one class entry rather than five overrides.

Four region classes and the tolerance each one earns Four region classes are listed with their per-pixel tolerance and the reason for it. Solid fills take the tightest tolerance because a flat colour cannot legitimately move between runs. Line work takes a moderate one to allow for join and cap rasterisation. Label glyphs take the loosest because hinting genuinely moves subpixels. Raster and hillshade imagery takes a moderate one for resampling differences. A caption states the consequence of collapsing these into one number: the single value has to be set for the loosest class, so glyph noise ends up licensing an equivalent change in a flat fill that occupies most of the frame. fills 0.0001 a flat colour cannot legitimately move lines 0.0005 join and cap rasterisation only labels 0.0030 glyph hinting genuinely moves subpixels raster 0.0008 resample differences on hillshade and imagery collapse these to one number and glyph noise licenses the same change in a flat fill

4. Constrain the values in schema so a percentage cannot be mistaken for a fraction 

{
  "changedPixelRatio": { "type": "number", "minimum": 0, "maximum": 0.05 },
  "ssimFloor":         { "type": "number", "minimum": 0.9, "maximum": 1 },
  "required":          ["reason", "owner", "review"]
}

The upper bound on changedPixelRatio is the important one: 0.3 meaning “0.3 percent” is a thousand-fold looser gate that passes everything, and it is the single most common profile bug.

5. Write one resolver, and publish what it resolved

Resolution order, printed with every result A capture arrives carrying three attributes: its zoom, its region class and its scenario identifier. The resolver applies the profile in four layers, each narrowing the last: the suite default, then the zoom band, then the region class, then any scenario override. The output is an effective threshold object plus a provenance list naming the layers that contributed. An arrow shows that list being written into the published result record alongside the measurement, so a surprising verdict is explained by one line of output rather than by reading the profile. The provenance list is what makes a verdict explicable capture zoom, region class, scenario id defaults zoom band region class scenario effective threshold + provenance changedPixelRatio 0.0008 · ssimFloor 0.991 resolvedFrom: defaults, zoomBand, region, scenario written into the result record beside the measurement, on every capture

6. Delete the scattered sources, then start reviewing

With the profile authoritative and the records proving what it produces, remove the inline options and environment variables. Only then begin tightening: take the migrated entries in order of looseness, measure each one’s p99 from the telemetry the records are now producing, and either justify the value or replace it with a class.

7. Schedule the audit that keeps the numbers falling

A monthly job compares each class’s configured tolerance against its measured p99 and opens a pull request where headroom has opened up. Without it, tolerances only ever rise, because raising one unblocks a release and lowering one creates work.

8. Decide what the profile deliberately does not cover

A profile that tries to express everything becomes unreadable, and three things are better kept out of it.

Masking rules. They look like configuration and they belong in the masking manifest, because their consumers are different: the capture runner, the comparator and the review interface all read masks, while only the comparator reads thresholds. Merging them produces a file that three systems parse for different reasons, and a change to one concern forces a review by owners of the other.

Scenario definitions. Which cameras exist, what they are called and what they load is test data, not gate configuration. Keeping them separate means adding a scenario does not touch the file that governs how strictly every scenario is judged — which matters because the two changes want different reviewers.

Retry and quarantine policy. How many times a flaky capture is retried and when it is quarantined belongs with the triage machinery described in Flaky Visual Test Triage. A tolerance answers “how much change is acceptable”; a retry budget answers “how much nondeterminism is acceptable”, and conflating them is how a team ends up raising a threshold to fix a flake.

The boundary that keeps all three straight is simple: the profile holds the numbers a comparison is judged against, and nothing else. Anything that changes what is captured, what is excluded, or what happens after a verdict lives elsewhere. A file with that boundary stays reviewable at a few hundred lines; one without it stops being read within a year, which returns the suite to the state the migration was meant to fix.

Verification

Confirm the procedure worked before wiring it into a blocking gate:

Troubleshooting

Symptom Likely cause Fix
The gate passes everything after the migration A tolerance was expressed as a percentage where the comparator expects a fraction Add the schema upper bound from step 4 — it catches this class of bug at validation rather than in production
A scenario’s verdict does not match its profile entry An inline option or environment variable is still winning over the resolver Delete the scattered source; if an emergency override is genuinely needed, commit it as a scenario entry with a review date
The audit proposes tightenings that immediately cause failures The p99 was computed over too short a window, or over runs from a single runner class Widen the window and ensure it spans every runner class in the fleet before treating a percentile as representative

Frequently asked questions

YAML or JSON for the profile?

Either, provided it is schema-validated. YAML wins on comments, which matter here because the reason field is doing real work and adjacent explanatory comments are common. JSON wins on tooling and on eliminating a class of indentation surprises. The decision that matters more than the format is that the file is data, read by one resolver, rather than a module that exports computed values.

Should thresholds be scoped per engine?

Yes if the suite runs more than one engine, because a tolerance calibrated against Chromium’s rasteriser is not valid for WebKit’s. The cheapest way is an engine key at the same level as the zoom band, resolved between the band and the region class. A shared profile across engines is in practice calibrated for the noisiest of them, which means the others are running looser than they need to.

How long should a migrated entry keep its "not yet reviewed" reason?

One audit cycle. Set the review date a month out at migration time, and let the audit report the expiries. The point is not that a month is enough to review everything, but that the expiry forces the list to be looked at rather than forgotten — and a migrated entry with a passed review date is a much clearer prompt than a comment nobody reads.

What if a scenario genuinely needs a unique tolerance?

Then it gets one, with a measured justification and a review date. The rule of thumb is that more than about five percent of scenarios having their own override means the region classes are wrong: each override claims that this capture is unlike every other capture in its class, and when many captures make that claim they are describing a class that does not exist yet.