Triaging a Large Map Baseline Re-Bless

The first genuinely large re-bless is where most visual suites are decided. A cartographer changes a road casing colour, three hundred and forty captures go red, and the team either finds a way to review that in twenty minutes or learns that approving in bulk is survivable — and once they have learned that, the suite has stopped asserting anything. This procedure is the twenty-minute version: group the failures by what caused them, review in the order that surfaces the unintended change, and leave behind a record that says what was actually judged.

This is a task within Baseline Review & Approval Workflows, under CI/CD & Visual Testing Operations. It assumes the gate published per-capture records with thresholds and provenance, per Visual Gate Threshold Configuration in CI.

Prerequisites

Step-by-step procedure

1. Compute a signature per failing capture

Four features from the diff you already have are enough to separate the common cases. Round them, concatenate, and group by equality.

function signature(diff) {
  const coverage = Math.round(diff.changedRatio * 200) / 200;   // 0.5% buckets
  const box = diff.largestCluster;
  const locality = box.w * box.h > 0.25 ? 'spread' : 'local';
  const delta = diff.meanDelta.map((v) => Math.sign(v) * Math.round(Math.abs(v) / 8));
  return [coverage, locality, delta.join(','), diff.dominantRegion].join('|');
}

The bucket widths matter more than the features. Too fine and a frame-wide change fragments into forty groups; too coarse and a dropped label hides inside the group that explains the colour shift. Start coarse and narrow only if genuinely different causes are landing together.

2. Rank the groups by strangeness, not by size

A 340-capture re-bless collapsed into four signature groups One failing run is grouped by diff signature. The largest group holds 312 captures sharing a uniform road-casing colour shift, which matches the style edit under review and is accepted as one batch. A second group of 19 captures shares a label-halo change, also explained by the edit. A third group of 6 captures shows a dropped label at high zoom, which the edit does not explain and which is investigated. A final group of 3 captures matches nothing else in the run and is sorted to the top of the review, because an ungrouped failure during a large re-bless is the most likely place for an unintended change to be hiding. Sort the ungrouped to the top, not the bottom 312 · uniform road-casing colour shift explained by the style edit — accept as one batch 19 · label halo change also explained — second batch 6 · dropped label at z16 not explained — investigate 3 · ungrouped review these first 92% 6%

A group of three that matches nothing else in the run is the highest-value thing in the review. It is where a change that nobody intended will be, because an unintended change almost never shares a signature with the intended one.

3. Review in the order that finds it

The order a large re-bless is reviewed in Four review stages run in a deliberate order. First the ungrouped failures are examined individually, because a large batch is where an unintended change hides most easily. Second the groups that the change under review does not explain are examined, since an unexplained signature is a finding whether or not it is large. Third the explained groups are confirmed against the release notes or the diff, one judgement per group. Only fourth is anything approved, and the approval records the group signature so the audit trail shows one decision rather than three hundred. A caption warns that reversing the first and third stages is what produces approve-all. Review smallest-and-strangest first, approve last 1 · ungrouped failures, individually where an unintended change hides 2 · groups the change does not explain a finding regardless of size 3 · explained groups, one judgement each confirm against the notes or the diff 4 · approve, recording the group signature one decision in the trail, not three hundred

4. Confirm each explained group against the change, out loud

For each large group, write one sentence in the pull request saying what the group is and why the change under review produces it. This takes a minute per group and is what makes the review auditable: a reviewer who cannot write that sentence has not actually confirmed anything, and the act of trying is what surfaces the group that does not quite fit.

5. Route what is not yours

A cartographic group goes to whoever owns the style; an interface group goes to the team that owns that panel; an environment group goes to whoever owns the runner image. A group that nobody owns is a finding, not an approval — route it to an investigation rather than to a person with permissions.

6. Approve as batches, and record the batch

What the trail records for a batch approval, and why each field is needed later A batch approval record carries five fields. The group signature identifies which cluster of captures was approved together, so the trail shows one judgement rather than many. The count records how many captures the judgement covered. The representative names the specific capture the reviewer actually looked at. The cause records what the group was attributed to, such as a named style edit or a library version. The environment digest records the browser, fonts and GL backend at the time. A caption states the test these fields exist to pass: an engineer a year later, asked whether a baseline can be trusted, should not need a CI log that has already expired. Five fields, so the batch is still explicable in a year groupSignature one judgement, not three hundred count how much the judgement covered representative the capture the reviewer actually saw cause style edit, library version, font package environment browser, fonts, GL backend, image digest CI logs expire at ninety days; the trail has to outlive them

7. Leave the unexplained open

A re-bless that lands with three captures still under investigation is a healthy outcome, and far better than one where those three were approved to make the branch green. Quarantine them, per Flaky Visual Test Triage if they turn out to be nondeterministic, or file them as bugs if they reproduce. What matters is that they leave the review as open questions rather than as new baselines.

8. Feed what you learned back into the signature

Every large re-bless teaches the grouping something, and capturing that is what makes the next one cheaper.

Three lessons recur. A cause that fragmented across several groups means a bucket is too narrow for that kind of change — widening it is a one-line edit and it compounds, because the same cause will recur. A cause that hid inside a larger group means a feature is missing; the usual culprit is that the signature has no term distinguishing where in the frame the change sits, and adding the largest-cluster box fixes an entire class of hiding. And a cause that recurs across runs — the font package, the browser bump, the quarterly fixture refresh — deserves a named label so the next occurrence is recognised on sight rather than re-derived.

Recording those as a short list beside the signature function, with a line per lesson and the run it came from, keeps the function from being tuned in circles. It is common for a team to widen a bucket after one painful re-bless, narrow it again after the next, and end up back where they started, because neither change was written down with its reason.

The broader point is that the signature is part of the suite’s machinery and deserves the same treatment as the thresholds: derived from what actually happened, changed deliberately, and recorded with a reason. A grouping function nobody owns degrades the same way an unowned tolerance does, just less visibly — the failures still group, they simply stop grouping usefully, and the review quietly gets longer until somebody starts approving in bulk again.

Verification

Confirm the procedure worked before wiring it into a blocking gate:

Troubleshooting

Symptom Likely cause Fix
A frame-wide change fragments into dozens of groups The coverage bucket is too narrow, so captures with slightly different changed ratios separate Widen the bucket — half a percent is a reasonable starting point — and drop the colour-delta term if it is still fragmenting
A dropped label hides inside the group explaining the colour shift The signature has no locality term, so a local blob and a frame-wide wash score alike Include the largest-cluster box as in step 1; locality is the feature that separates these two
Every group routes to the same person Scenarios have no ownership metadata, so routing falls back to a default reviewer Add ownership to the scenario definitions; a new scenario should not be addable without someone deciding who owns it

Frequently asked questions

How long should a 340-capture re-bless take with this procedure?

Twenty to thirty minutes for a reviewer who knows the style. Most of that is the ungrouped failures and the one group that does not quite fit; the large explained groups are a minute each. If it is taking hours, the signatures are fragmenting — widen the buckets before concluding that the change is simply large.

Should the grouping be shown to the reviewer, or used to auto-approve?

Shown. Grouping protects the reviewer’s attention; auto-approving replaces their judgement, which is the only part of the process doing anything. The distinction matters because the two look similar in a tool’s interface: a system that groups and asks is doing the right thing, and one that groups and decides has quietly removed the gate.

What if the change under review genuinely explains everything?

Then the review is four sentences and a batch approval, which is the correct outcome and takes minutes. The value of the procedure is not that it always finds something; it is that when something is there, it is found rather than approved. A run where everything is explained is a good run, provided somebody actually checked rather than assumed.

Can signatures be reused across runs to recognise known causes?

Yes, and it is the natural next step: a signature seen before and previously attributed to a font package update can be labelled automatically on sight, which turns an environment re-bless into a one-click routing decision. Keep the label as a suggestion rather than an approval, for the same reason as above.