Baseline Review & Approval Workflows

The technical half of a visual suite ends when a diff is produced. The half that decides whether the suite is worth anything begins there, and it is a human process: someone has to look at 340 changed captures after a style edit and say which of them are the intended change. Get that process wrong and the outcome is always the same — the reviewer clicks approve-all, the baselines update, and the suite has recorded a change rather than checked one. This page is about the workflow that keeps the judgement real: how the diffs are batched, how they are presented, who is asked, and what is left behind afterwards.

This page belongs to CI/CD & Visual Testing Operations and picks up where Visual Gate Threshold Configuration in CI leaves off: the gate has made a decision, published its numbers, and now a person has to act on it.

Why approve-all happens, and what actually prevents it

Nobody sets out to rubber-stamp. It happens because of a specific mismatch: the reviewer is shown a list of captures when the thing they can reason about is a list of causes.

A style edit that changes the road casing colour touches every capture containing a road. Presented as 340 individual diffs, the reviewer’s task is to look at 340 near-identical pictures and confirm each one — which is not a judgement, it is an endurance test, and the rational response after the twentieth is to approve the rest. Presented as one group of 340 captures sharing a single diff signature, with three representative examples and a note that the remaining 337 match the same signature, the task is one judgement about one change, which is exactly what the reviewer is qualified to make.

Grouping is therefore not a convenience feature; it is the mechanism that makes review possible at all. Everything else in this page follows from it.

The same 340 failures presented as captures and as causes Two presentations of one failing run. On the left, 340 individual diff thumbnails are shown as a dense grid, and the caption notes that the reviewer's only realistic action is to approve them in bulk. On the right, the same failures are grouped into three clusters by diff signature: 337 captures sharing a uniform road-casing colour shift, two captures with a dropped label, and one capture with an unmasked timestamp. Each group carries a count and a representative example. The caption states the difference: three judgements a cartographer can make, instead of one decision they cannot. Group by cause, not by capture 340 diffs 3 causes the only realistic action is approve-all 337 · uniform road-casing colour shift matches the style edit under review 2 · dropped label at z16 not explained by the edit — investigate 1 · unmasked timestamp a masking gap, not a regression three judgements a cartographer can actually make

Grouping failures by signature

The grouping does not need to be clever. Four cheap features, computed from the diff the comparator already produced, separate the common cases well enough to be useful.

Coverage — what fraction of the frame changed. Frame-wide changes group together and almost always share a cause.

Locality — the bounding box of the largest connected diff cluster, normalised to the frame. A change confined to one small box is a different kind of event from one scattered across the frame.

Colour delta — the mean signed change per channel across changed pixels. A colour ramp edit produces a consistent signed delta; a dropped feature produces a delta dominated by the background colour.

Region class — which of the tolerance classes the changed pixels fall into, from the region map already used for thresholding.

Rounding those four into a short signature and grouping by equality is enough to collapse a style-wide change into one row.

function signature(diff) {
  const cov = Math.round(diff.changedRatio * 200) / 200;       // 0.5% buckets
  const box = diff.largestCluster;
  const locality = box.w * box.h > 0.25 ? 'spread' : 'local';
  const dc = diff.meanDelta.map((v) => Math.sign(v) * Math.round(Math.abs(v) / 8));
  return [cov, locality, dc.join(','), diff.dominantRegion].join('|');
}

Groups of one are as informative as groups of three hundred: a single capture whose signature matches nothing else in the run is the one most likely to be an unintended change riding along with an intended one, and it should be sorted to the top of the review rather than buried in a list.

Presenting a diff a cartographer can judge

Three views, and the ability to move between them quickly, cover essentially every question a reviewer asks.

Side by side answers “what does the new one look like”. It is the view people expect and the least useful for spotting small changes, because the eye cannot register a two-pixel shift across a gap.

Onion skin / swipe answers “what moved”. A slider that wipes between baseline and candidate in place makes a subpixel label shift obvious in a way no other presentation does, because the change becomes motion rather than difference.

Diff mask answers “where did it change”, and is the one that should be shown first in a grouped review, because it is the view in which the signature is legible — a uniform wash, a single blob, a traced outline.

Two presentation details matter more than they sound. Excluded regions must be drawn on the diff view, or a reviewer will spend time wondering why an obviously changed area shows no difference. And the effective threshold and measured value should sit next to the image, because “0.31% against a budget of 0.04%” tells a reviewer how far outside normal this is, which a picture does not.

Three review views and the question each one answers well Three presentations of the same failing capture are shown side by side. The side-by-side view places baseline and candidate next to each other and answers what the new frame looks like, but hides small shifts across the gap between them. The swipe view overlays them in place with a movable divider and answers what moved, making a subpixel label shift visible as motion. The diff mask view shows only the changed pixels and answers where the change is, which is the view in which a group's signature — a uniform wash, one blob, a traced outline — can be recognised at a glance. A note says the mask should be shown first in a grouped review. Show the mask first; it is the view a signature is legible in side by side swipe diff mask what does it look like? hides small shifts what moved? turns difference into motion where did it change? the signature is visible here draw the excluded regions on the mask, or a reviewer wonders why a changed area shows nothing

Who approves what

A single approver list for all baseline changes produces either a bottleneck or a rubber stamp. Routing by what changed is both faster and more careful.

  • Cartographic changes — colour, typography, symbology, label placement — go to whoever owns the style. They are the only people who can say whether the new rendering is the intended one, and they will spot a wrong shade that an engineer reads as “the colour changed, as expected”.
  • Application changes — a control moved, a panel resized, an overlay’s layout — go to the team that owns that interface.
  • Environment changes — a browser bump, a font package update, a GL backend change — go to whoever owns the runner image, and should be blessed as a batch with the environment change recorded in the annotation, because attributing them to whatever pull request happened to be open is how the provenance is lost.
  • Anything unexplained goes to nobody automatically. A group whose signature does not match the change under review is a finding, and routing it to an approver invites it to be approved.

Codifying this as ownership metadata on the scenarios — the same file that carries the tolerance overrides — means the routing is data rather than convention, and a new scenario cannot be added without someone deciding who owns it.

The audit trail, and what it has to survive

The trail is worth designing for a specific future event: an engineer who was not present, six months later, looking at a baseline and asking whether it can be trusted. Four fields answer that, and they belong on the artifact rather than in a chat log.

Field Answers
approvedBy Who accepted it, resolvable to a person after they change teams
approvedAt When, in a form that can be compared against the environment history
commit What change it was blessed for, linking to the review that discussed it
groupSignature Which group it was approved as part of, so a bulk approval is visible as one
supersedes The baseline it replaced, so the chain back to the original is walkable
environment Browser, GL backend, font package and image digest at the time

The last two carry the most weight in practice. supersedes turns a baseline store into a history rather than a snapshot, so a suspicious frame can be walked backwards until the run that introduced the change. environment is what lets someone distinguish “this was blessed when we were on the old font package” from “this was blessed deliberately”, which is otherwise unanswerable and is the single most common question asked of an old baseline.

Storing the trail beside the artifact rather than in the CI provider matters because CI logs expire. A ninety-day retention policy is entirely normal and quietly makes every baseline older than ninety days unattributable.

Measuring whether the review is real

A review process can look healthy and be doing nothing, so it is worth instrumenting. Four numbers, all derivable from the audit trail, say whether judgement is actually happening.

Rejection rate. The share of proposed baseline changes that a reviewer declines. A process with a rejection rate of zero over a quarter is not catching anything, and the most likely explanation is that reviewers are approving whatever they are shown. A healthy suite rejects something every few weeks — usually an unintended change that rode along with an intended one.

Time from proposal to decision, by group size. If large groups are decided faster than small ones, reviewers are spending less attention on more change, which is the signature of approve-all reasserting itself through the grouping mechanism rather than despite it.

Ungrouped share. The fraction of failing captures whose signature matched nothing else in the run. This should be small and non-zero. A run where everything grouped means the signatures are too coarse to separate an unintended change from the intended one; a run where nothing grouped means they are too fine to help.

Approver concentration. The share of approvals made by the single most active approver. A number above about sixty percent means the routing is not working — everything is reaching one person, who is by now approving on trust rather than on knowledge of the style.

Four review health metrics and the failure each one exposes Four metrics are listed with a healthy range and the specific failure an unhealthy value indicates. A rejection rate of zero over a quarter indicates that reviewers are approving whatever they are shown. Large groups being decided faster than small ones indicates approve-all operating through the grouping mechanism. An ungrouped share of zero indicates signatures too coarse to separate an unintended change from an intended one, while a share near one indicates signatures too fine to help. Approver concentration above about sixty percent indicates routing that is not working, with everything reaching one person who is approving on trust. Four numbers that say whether judgement is happening rejection rate healthy: something declined every few weeks zero for a quarter → reviewers approve whatever they are shown decision time by group size healthy: rises with the size of the group falls with size → approve-all, now wearing the grouping as cover ungrouped share healthy: small and non-zero zero → signatures too coarse · near one → too fine to help approver concentration healthy: below about 60% for any one person above it → routing has collapsed onto one reviewer

None of these require new instrumentation: every one is a query over the fields the audit trail already carries. Publishing them on the same dashboard as the suite’s pass rate keeps the review process visible as a thing that can degrade, rather than as a step that either happened or did not.

Common pitfalls

Reviewing in the CI log. Diff images linked from a job’s artifacts are three clicks and a download away from the person deciding, which is enough friction to guarantee they decide without looking. The review surface belongs where the code review is.

Approving during an incident. A release is blocked, the gate is red, and someone with permissions approves the baselines to unblock it. Nothing about that is unreasonable in the moment, and it permanently destroys the meaning of those baselines. The right emergency valve is a documented, expiring override of the gate — as a committed profile entry — not a permanent rewrite of the reference.

No distinction between a new baseline and a changed one. A scenario that has never had a baseline needs someone to look at it once and say “yes, that is what this should look like”. A scenario whose baseline is being replaced needs someone to say “yes, that change is intended”. Presenting both as “approve?” gets the first one waved through, and a first baseline that was never really looked at is a permanent record of a bug.

Approver lists that grew rather than were designed. Every team that has ever needed to unblock a release ends up on the list, and after a year the set of people who can rewrite the map’s reference images is larger than the set who can deploy it.

Treating the review queue as a backlog. A queue that is allowed to accumulate stops being reviewed at all, because the cost of entering it grows with its depth and the oldest entries are the least remembered. The queue should be drained per pull request, not per sprint, and a run whose failures are not decided within a working day should block the branch rather than wait — which is the same reasoning that makes the gate required in the first place.

Losing the group. If the bulk approval of 337 captures is recorded as 337 independent approvals, the audit trail says a person examined 337 frames, which is false. Recording the group signature keeps the record honest about what was actually judged.

Frequently asked questions

How long should a baseline review take for a large style change?

With grouping, minutes rather than hours — that is the point of it. A style change that touches every capture should collapse into a small number of signatures, and the reviewer’s work is to confirm each signature matches the intended edit and to look properly at whatever did not group. If a review still takes hours after grouping, the signatures are too fine-grained; widen the buckets until the common case is one row.

Should approval be required for every baseline, including new scenarios?

Yes, and new scenarios deserve more attention than changes, not less. A replaced baseline has a predecessor a reviewer can compare against; a first baseline has nothing, so the only check on it is someone looking at the frame and confirming it is correct. A first baseline approved without that look becomes a permanent reference for whatever state the application happened to be in.

What about baselines that change because the environment changed?

Bless them as one batch, attributed to the environment change rather than to a feature branch, with the image digest recorded in the annotation. This keeps the provenance readable: a year later the store shows “these 900 baselines moved when we upgraded the font package”, which is a fact someone can act on, rather than 900 baselines attributed to unrelated pull requests.

Can any of this be automated?

The grouping, the routing and the presentation should all be automated; the judgement should not. A classifier that recognises a known environment signature and files it as such is doing exactly the kind of work that protects a reviewer’s attention. A classifier that decides an unexplained change is probably fine has replaced the only part of the process that was doing anything, which is the same failure as approve-all with more steps.

How does the review workflow interact with quarantined flaky tests?

They should never meet. A capture that differs from itself between runs is not a baseline decision at all, and putting it in a review queue asks a person to judge a picture that has no stable answer. Route it to the flake queue described in Flaky Visual Test Triage, and keep the review queue for captures whose difference is reproducible.