Visual Gate Threshold Configuration in CI

A visual gate is only as good as the numbers it enforces, and in most pipelines those numbers live in three or four different places: a default in the comparator’s options, an override in a test file, an environment variable someone added during an incident, and a continue-on-error that quietly disables the whole thing for one job. The result is a gate nobody can reason about, where the answer to “what tolerance is this scenario actually running at” requires reading the pipeline. This page is about making that answer readable: one resolution order, one file, and a failure contract the pipeline publishes rather than infers.

This page belongs to CI/CD & Visual Testing Operations and is the pipeline-side companion to Dynamic Threshold Configuration: that page decides what the numbers should be, this one decides where they live and how a run resolves them.

What a gate configuration has to answer

Before choosing a file format it is worth being precise about the questions a threshold configuration exists to answer, because a layout that answers all five is barely more work than one that answers two.

  1. What tolerance applies to this capture? Not to the suite — to this scenario, at this zoom, in this region class, on this engine. Everything else is a default that happens to apply.
  2. Where did that number come from? A tolerance without a provenance is a guess with a decimal point, and the first time it blocks a release someone will change it rather than defend it.
  3. Who may change it, and did they? A tolerance that can be raised by an environment variable in a pipeline file is a tolerance with no owner.
  4. What does the gate do when it trips? Fail the job, comment on the pull request, upload the diff, or all three — and whether the check is required is part of the configuration, not a repository setting nobody reads.
  5. Is the gate on at all? The most common way for a visual gate to stop working is not a wrong number; it is a job that silently skipped, or a continue-on-error: true added during an outage and never removed.
The resolution order that turns four scattered settings into one answer A resolution chain runs from the most general setting to the most specific, with each layer narrowing the one before it. The suite default sits at the base. A per-zoom band narrows it for the capture's zoom. A per-region class narrows it further for the part of the frame being scored. A per-scenario override, used rarely and requiring a recorded reason, is the most specific. To one side, two settings are drawn outside the chain and crossed out: an environment variable and a job-level continue-on-error, both of which change the gate's behaviour without appearing anywhere in the resolution. The caption states the property that matters — a single function of the capture's own attributes produces the number, so it can be printed alongside the result. One resolution order, printable next to every result suite default per-zoom band per-region class per-scenario override env var override changes the gate, appears nowhere continue-on-error disables it entirely, silently if the effective tolerance cannot be printed next to the result, the gate cannot be reviewed

Architecture: one profile, resolved in code

The configuration is a single file, version-controlled beside the tests, and a small resolver that turns a capture’s attributes into an effective threshold object. Nothing else reads or writes thresholds.

# visual-gate.yml
version: 3
defaults:
  changedPixelRatio: 0.0006
  ssimFloor: 0.990
  maxClusterPx: 400

zoomBands:
  - { range: [0, 4],   changedPixelRatio: 0.0010, ssimFloor: 0.985 }
  - { range: [5, 10],  changedPixelRatio: 0.0006, ssimFloor: 0.990 }
  - { range: [11, 15], changedPixelRatio: 0.0004, ssimFloor: 0.993 }
  - { range: [16, 22], changedPixelRatio: 0.0003, ssimFloor: 0.995 }

regionClasses:
  fills:   { changedPixelRatio: 0.0001, note: "a flat fill cannot legitimately move" }
  lines:   { changedPixelRatio: 0.0005 }
  labels:  { changedPixelRatio: 0.0030, note: "glyph hinting; measured p99 was 0.0019 on 2026-06-14" }
  raster:  { changedPixelRatio: 0.0008 }

scenarios:
  harbour-night-z17:
    ssimFloor: 0.991
    reason: "hillshade dithering under the night ramp; p99 0.9934 over 60 runs"
    owner: "@cartography"
    review: 2026-11-01

gate:
  required: true
  onFail: [upload-diff, comment-pr, fail-job]

Three properties of this layout are load-bearing.

Every override carries a reason, an owner and a review date. A scenario entry without them fails schema validation, which is the mechanism that stops an incident-time override becoming permanent. The review date is what the monthly audit reads.

Region classes hold the measured percentile in the note. Six months later, deciding whether labels can be tightened is a comparison against a recorded measurement rather than an argument.

The gate’s behaviour is in the same file as its numbers. Whether the check is required, and what it does on failure, are properties of the gate, and keeping them next to the thresholds means one file review shows the whole contract.

// resolver — the only code that reads the profile
export function resolveThreshold(profile, capture) {
  const band = profile.zoomBands.find(
    (b) => capture.zoom >= b.range[0] && capture.zoom <= b.range[1]
  );
  const region = profile.regionClasses[capture.regionClass] ?? {};
  const scenario = profile.scenarios[capture.scenario] ?? {};
  const effective = { ...profile.defaults, ...band, ...region, ...scenario };
  return {
    ...effective,
    resolvedFrom: [
      'defaults',
      band && `zoomBand:${band.range.join('-')}`,
      capture.regionClass && `region:${capture.regionClass}`,
      profile.scenarios[capture.scenario] && `scenario:${capture.scenario}`,
    ].filter(Boolean),
  };
}

resolvedFrom is the small detail that makes the whole thing reviewable: every result the gate publishes carries the list of layers that produced its threshold, so a surprising pass or fail is one line of output away from an explanation.

What one published gate record contains, and which question each field answers A single result record is decomposed into four groups of fields with the question each answers. The identity group — scenario, engine and region class — answers which capture this is. The measured group — changed-pixel ratio and SSIM — answers what the comparison found. The threshold group answers what it was judged against. The provenance group, resolvedFrom, answers where that threshold came from, listing the profile layers that contributed. A footer notes that the provenance field is what turns a surprising verdict into a one-line explanation instead of an investigation. Four groups, four questions, one line of output scenario · engine · regionClass which capture is this? the identity a reviewer searches by measured.changedPixelRatio · measured.ssim what did the comparison find? the observation, independent of any judgement threshold.changedPixelRatio · threshold.ssimFloor judged against what? the effective numbers, after resolution resolvedFrom: [defaults, zoomBand:16-22, region:raster, scenario:…] where did it come from? the layers that contributed, in order

Step-by-step: wiring the profile into a pipeline

1. Validate the profile before anything else runs. A schema check as the first step of the job costs half a second and converts a malformed override — a missing reason, a ratio expressed as a percentage rather than a fraction — into a fast, clear failure instead of a confusing gate result.

- name: Validate visual gate profile
  run: npx ajv validate -s schema/visual-gate.schema.json -d visual-gate.yml

2. Resolve and publish the threshold with every result. The comparator writes one record per capture, and the record carries the effective numbers and their provenance alongside the measurement.

{
  "scenario": "harbour-night-z17",
  "engine": "chromium-121",
  "regionClass": "raster",
  "measured": { "changedPixelRatio": 0.00071, "ssim": 0.9946 },
  "threshold": { "changedPixelRatio": 0.0008, "ssimFloor": 0.991 },
  "resolvedFrom": ["defaults", "zoomBand:16-22", "region:raster", "scenario:harbour-night-z17"],
  "verdict": "pass"
}

3. Fail on the worst region, never on an average. The job’s exit status is decided by whether any single record failed, for the reason set out in Diff Algorithm Tuning for Cartography: a large calm region will otherwise outvote a small broken one.

4. Make the check required, and prove it. Marking a check required is a repository setting, which means it is invisible in the diff and easy to lose. Add a test that asserts the gate actually blocks: a scheduled job that opens a draft pull request containing a deliberately broken baseline and asserts the merge is blocked. It runs weekly, costs a minute, and is the only thing that catches a required check that was quietly demoted.

5. Publish the summary where the reviewer already is. A comment on the pull request with the failing scenarios, their measured values, their thresholds and the provenance list — not a link to a log. The gate’s job is to make a decision legible, and a reviewer who has to open a CI log to see which number was exceeded will eventually stop looking.

6. Run the audit monthly. Compare each region class’s configured tolerance against its measured p99 across green runs, and open a pull request proposing anything that has more headroom than it needs. This is the ratchet that keeps tolerances falling rather than climbing.

Thresholds drift upward unless something pushes back Two trajectories of a region class's tolerance over a year. Without an audit, the tolerance only ever rises: each incident raises it to unblock a release, and nothing ever lowers it, so after twelve months it sits several times looser than the measured noise justifies and the gate detects very little. With a monthly audit, the same incidents still raise it, but each month the audit compares the configured value against the measured ninety-ninth percentile and proposes a tightening where headroom has opened up, so the line ratchets downward between incidents and ends the year tighter than it began. Without a ratchet, a tolerance has only one direction no audit monthly audit looser tighter JanAprAugDec both lines see the same incidents — only one of them ever gives the headroom back

Cross-environment considerations

  • Thresholds are per-engine whether or not you write them down. A tolerance calibrated on Chromium with SwiftShader is not valid for WebKit, whose glyph rasterisation is its own. Either scope the profile by engine or run only one engine; a shared profile across engines is a profile calibrated for the noisiest of them.
  • Runner class changes the noise floor. A move from a four-core to a two-core runner does not change rendering, but it does change how often a capture lands near a timing edge, which shows up as more captures near the top of their tolerance. Re-measure after a runner change rather than assuming.
  • Self-hosted runners drift. A fleet where images are updated independently produces a noise floor that varies by machine. Pinning the image digest, as described in Containerized Rendering Environments for Map Tests, is a prerequisite for a stable threshold rather than an optimisation.
  • Forks and pull requests from outside typically cannot read repository secrets, so a gate that needs a baseline store credential will skip rather than fail. Decide explicitly whether that is acceptable, and make the skip visible in the check’s output.

Migrating an existing pipeline onto a profile

Most teams arrive at this page with thresholds already scattered, and the migration is worth doing in a specific order so the suite never goes unguarded.

Inventory first, change nothing. Grep for every place a tolerance is set: comparator options, test files, environment variables, provider-level variables, and any continue-on-error or if: always() that softens a job. Write them into the profile as scenario entries with their current values and a reason of “migrated, not yet reviewed”. The suite behaves identically at this point, which is the property that makes the step safe.

Add the resolver and the published record next. Still without changing a number. Now every result carries its effective threshold and provenance, and for the first time the team can see the shape of what they have — usually that a handful of scenarios are running at tolerances ten times looser than the default, and that nobody remembers why.

Delete the scattered sources. With the profile authoritative and the records proving it, remove the inline options and the environment variables. This is the step that actually changes behaviour, and because the previous step published the effective values, any difference is immediately visible.

Then start reviewing. Take the migrated entries in order of looseness, measure each one’s p99 from the telemetry the records are now producing, and either tighten it with a recorded justification or promote it into a region class if several scenarios share the same reason. A team doing this typically eliminates two thirds of the scenario overrides in the first pass, because most of them turn out to be the same problem written down four times.

The reason for this order is that each step is independently revertible. A migration that changes the file layout and the numbers at the same time produces a suite whose results differ for two reasons at once, and the first surprising failure sends someone back to the old configuration wholesale.

Configuration reference

Key Type Notes
defaults.changedPixelRatio fraction Always a fraction, never a percentage — the single most common profile bug
defaults.ssimFloor 0–1 Applied per region, compared against the region’s mean, never the frame’s
defaults.maxClusterPx integer Largest connected diff blob permitted; catches localised faults a ratio misses
zoomBands[].range [min, max] inclusive Bands should be cut at content transitions, not at even intervals
regionClasses.<name>.note string Records the measured percentile and the date it was measured
scenarios.<id>.reason string, required Schema-enforced; an override without one fails validation
scenarios.<id>.review ISO date, required Read by the monthly audit; a passed date is reported as expired
gate.required boolean Documents intent; the weekly block test verifies reality
gate.onFail list upload-diff, comment-pr, fail-job — explicit rather than implied by job structure

Common pitfalls

Expressing a ratio as a percentage. changedPixelRatio: 0.3 meaning “0.3 percent” is a thousand-fold looser gate than intended and passes every test that has ever been written. A schema that constrains the value to a plausible range — say, below 0.05 — catches it at validation time.

Letting an environment variable win. It is convenient during an incident and it removes the profile’s authority permanently, because from then on the effective threshold is not knowable from the repository. If an emergency override is genuinely needed, make it a committed scenario entry with a reason and a review date one week out; the pull request takes a minute and leaves a trail.

Auto-blessing on failure. A pipeline that regenerates the baseline whenever the gate fails converts the gate into a recorder. It always looks like it is working, and it has never once refused anything. If baselines need frequent re-blessing, the problem is upstream determinism, not the approval step.

Thresholds that only exist in code. A tolerance passed inline to the comparator in a test file is invisible to the audit, cannot be resolved by the resolver, and will not appear in resolvedFrom. One file, one reader.

No test that the gate blocks. Every other item on this list produces a wrong number; this one produces no gate at all, and it is the most common failure in practice. The weekly block test is the only cheap defence.

Frequently asked questions

Should thresholds live with the tests or with the pipeline?

With the tests, in the repository, as data rather than code. The pipeline reads them; it does not own them. That placement means a threshold change goes through the same review as a code change, appears in blame, and reverts cleanly — none of which is true of a value set in a CI provider’s web interface, which is invisible in the repository and typically editable by a wider group than the code is.

How many scenario-level overrides is too many?

As a rule of thumb, more than about five percent of scenarios having their own override means the region classes are wrong. Each override is a statement that this one capture is unlike every other capture in its class; when many captures make that statement, they are describing a class that does not yet exist. Creating the class is both cheaper to maintain and more honest about what is being measured.

Can the gate be advisory while a suite is being built?

Yes, and it should be — for a bounded period, recorded in the profile. gate.required: false with a review date makes the temporary state explicit and gives the audit something to report when the date passes. The failure mode to avoid is an advisory gate with no end date, which accumulates red results nobody reads and eventually gets deleted for being noisy.

What should the gate do about a scenario with no baseline?

Fail, and say so distinctly from a diff failure. A missing baseline usually means a new scenario was added without blessing, or a key changed — a camera moved, a style hash rolled — and both need a human decision. Treating it as a pass creates a scenario that has never been checked and looks green; treating it as a diff failure sends the reviewer looking for a rendering change that does not exist.

How does this relate to per-region tolerance classes?

The region classes here are the pipeline-side expression of the same idea developed in Dynamic Threshold Configuration. That page derives what the numbers should be from measured telemetry; this one is about where the resulting numbers live, how a run resolves them, and how they are kept honest over time. The region map that assigns pixels to classes is shared between them.