Flaky Visual Test Triage & Quarantine

A flaky map visual test is one that returns a different verdict — pass or fail — on the same commit, the same baseline, and the same runner image, with nothing changed but time. In a slippy-map suite these are not rare: a tile that arrives one frame late, a label the collision engine placed in the opposite order, a GPU that dithered a gradient differently, or a marker cluster that expanded on a different animation tick all produce a diff that clears on the next run. Left unmanaged, flakes do more damage than a broken build — they train engineers to hit re-run on red, and a re-run reflex is exactly how a genuine regression slips through a gate that everyone has learned to ignore. This guide covers the operational discipline that keeps a noisy suite trustworthy: measuring flakiness numerically, quarantining a test without silently dropping its coverage, spending a retry budget without hiding real failures, and separating a determinism bug from a true regression before you touch a baseline.

This guide is part of CI/CD Integration & Visual Test Operations, the parent area covering how map visual tests run, gate, and scale inside a delivery pipeline; here we focus narrowly on the operational lifecycle of the tests that misbehave.

What “flaky” means, measured #

The intuitive definition — “fails sometimes” — is not actionable because it gives you no threshold at which to act. Formalize it as a rate. For a fixed commit, run the same test $N$ times and let $f$ be the number of runs whose verdict disagrees with the majority. The flakiness score is:

$$S=\frac{f}{N}$$

A test that passes 9 of 10 identical runs scores $S=0.1$; a coin-flip test scores $S\approx 0.5$. A deterministic test scores $0$. The value of a rate is that it sorts your suite: a test at $S=0.4$ is destroying your gate’s credibility and must be quarantined today, while one at $S=0.02$ is a background annoyance you schedule for later.

A single-commit sample is noisy, so track the score as an exponentially weighted moving average across the last runs the test has actually done in CI, discounting old evidence so a test that was fixed stops looking flaky:

$$S_t=\alpha \cdot x_t+(1-\alpha )S_{t-1},x_t\in \{0,1\}$$

Here $x_t=1$ when run $t$ disagreed with the run’s peers on the same commit (a reversal), and $\alpha \in [0.1,0.3]$ sets how fast history decays — a higher $\alpha$ reacts quickly to a newly flaky test but is jumpier. The key subtlety for maps: a plain pass/fail history cannot distinguish flake from regression, because a real break also flips the verdict. You only score a run as flaky ($x_t=1$) when the same commit produced disagreeing results — which is why the re-run-the-same-commit technique below is not just a diagnostic, it is the measurement itself.

Two derived quantities drive the dashboards later: flake rate, the fraction of the whole suite currently above a quarantine threshold, and mean time to resolution (MTTR), how long a test sits quarantined before it is fixed or deleted. Flake rate measures how noisy your gate is; MTTR measures whether your team actually clears the quarantine or just hides tests in it forever.

The quarantine lifecycle #

Quarantine is not “delete the test.” It is a controlled state in which a suspect test keeps running and keeps reporting, but its result no longer blocks the merge gate. Coverage is preserved — the test still captures frames, still diffs, still surfaces a signal on the dashboard — while the pipeline stops treating that signal as a veto. The lifecycle has five states, and the discipline is in the transitions, not the tags.

Detect is the automated entry point: a scheduled job re-runs recently changed or historically noisy tests against a pinned commit and updates each test’s score. Any test crossing the quarantine threshold is a candidate. Tag attaches machine-readable metadata — a @quarantine annotation, the triggering score, and a link to a tracking issue — so the state is visible in code review, not buried in a CI log. Isolate is the load-bearing move: the runner still executes the test and still uploads its diff, but the gate treats a quarantined failure as a warning, not a blocker. Track keeps the test on a dashboard with its live score and age, so a quarantined test cannot quietly become permanent. Finally the test leaves quarantine one of two ways: fix, when a root cause is addressed and the score falls back to zero, returning it to the gate; or delete, when a test guards a view no longer worth the flake it generates. A quarantine with no exit is technical debt with a green checkmark.

The one rule that makes this safe: a test may enter quarantine automatically, but it may only leave by a human decision recorded against the tracking issue. Automatic re-promotion invites a test that flickered quiet for a day to slide back into the gate still broken.

Retry budgets, and why blind retries are dangerous #

The tempting fix for flake is a blanket retry: configure the runner to re-run any failing test two or three times and pass it if any attempt succeeds. This is a trap. A retry-until-green policy cannot tell a flaky test from a genuinely broken one — a real regression that fails 100% of the time is indistinguishable, to the retry logic, from a flake that fails 30% of the time, right up until the regression fails all its retries and someone assumes that was flake too. Blind retries convert your gate from “does this pass?” into “did this pass at least once out of $k$?”, which is a strictly weaker guarantee, and they hide the flakiness signal you need to actually fix the problem.

Model the risk. If a test’s real per-run failure probability is $p$ and you allow $k$ attempts, the probability the gate reports green is:

$$P(\text{pass}\mid k)=1-p^k$$

For a true regression $p\to 1$, so even $k=3$ leaves $p^k$ close to 1 and the regression is still caught — good. But the same formula is why retries mask partial regressions: a real break that only manifests on 40% of runs ($p=0.4$) sails through at $k=3$ with a 94% chance of a green report. Retries do not distinguish “flaky and harmless” from “flaky because half-broken.”

So spend the retry budget deliberately, with three constraints:

1. Cap it and count it. Allow a small fixed number of retries (one, occasionally two) and record every retry as a flakiness event feeding the score. A test that only passes on retry is not passing — it is quarantine-bound.
2. Retry the whole test, not the assertion. Re-running just the pixel comparison against the same captured frame proves nothing; the flake is usually in capture, so a meaningful retry re-navigates, re-synchronizes, and re-captures.
3. Never retry silently. A retry that leaves no trace is a masked failure. Surface “passed on attempt 2 of 2” in the report so the trend is visible.

The full runner configuration for this — capping attempts, tagging retried tests, and auto-filing the quarantine issue — is worked through in auto-retrying and quarantining flaky map screenshot tests.

Separating a determinism bug from a real regression #

This is the central diagnostic skill, and for web maps it has a clean, mechanical test. When a diff fails, the question is never “is this a real change?” in isolation — it is “does the code under test produce a stable frame at all?” A determinism bug means two runs of the identical commit disagree with each other; a real regression means the commit reliably produces a frame that differs from the baseline. These are distinguished by re-running the same commit twice and diffing the two current captures against each other, ignoring the baseline entirely.

The logic is exhaustive. If the two current captures of the same commit differ from each other, the code cannot even reproduce its own output, so the baseline comparison is meaningless — you have a determinism bug and no amount of baseline re-blessing will fix it. If the two current captures are byte-identical (or SSIM-identical) to each other yet both differ from the baseline, the change is real and reproducible: either an intended visual change that should re-bless the baseline, or an unintended regression that should open a bug. This single test collapses the ambiguous “is it flake or a break?” question into two unambiguous branches.

For web maps, the unstable branch almost always resolves to one of a handful of causes, and each has an established fix elsewhere on this site. The largest single source is capture racing the renderer — the shutter firing before every tile has hydrated — addressed by Handling Async Tile Loading. Residual GPU and anti-aliasing variance that survives correct synchronization is absorbed by Noise Reduction for Map Artifacts. Moving UI — cursors, tooltips, animated markers, live traffic — is held still by the techniques in Dynamic Element Masking & UI Stability. And cross-engine rendering divergence, where the same commit renders differently on Chromium versus WebKit, is a determinism problem only if you share one baseline across engines; the fix is a per-engine baseline as laid out in the Cross-Browser Baseline Matrix.

Instrumenting the triage loop #

The following procedure wires the score, the re-run test, and the quarantine tag into a CI job. It assumes your suite already produces a per-test pass/fail plus a diff ratio, and that tests are annotatable (Playwright’s test.info().annotations, a tag convention, or a sidecar JSON).

1. Record every run’s verdict against its commit SHA. Persist a small event per test per run so the score can be computed over history. Keep it outside the repo — a CI artifact store or a tiny database — keyed on test id and commit.

   { "test": "home-z12@chromium", "sha": "a4f9c1e", "verdict": "fail",
     "diffRatio": 0.031, "attempt": 1, "runId": "8814", "ts": "2026-07-14T09:12:03Z" }
   

2. On a failure, re-run the same commit and self-diff. Capture the view twice from the identical build and compare the two current frames to each other before ever consulting the baseline. This both diagnoses and produces the flakiness signal $x_t$.

   const a = await captureView(page, view);   // first capture of this commit
   const b = await captureView(page, view);   // second capture, same commit
   const selfDiff = pixelmatch(a.data, b.data, null, a.width, a.height,
     { threshold: 0.1, includeAA: false });
   const unstable = selfDiff / (a.width * a.height) > 0.0005;
   // unstable === true  -> determinism bug (score it as flaky, x_t = 1)
   // unstable === false -> reproducible; compare to baseline for regression
   

3. Update the flakiness score. Apply the EWMA with the reversal signal from step 2, and compare against the quarantine threshold.

   const alpha = 0.2, QUARANTINE = 0.15;
   score[test] = alpha * (unstable ? 1 : 0) + (1 - alpha) * (score[test] ?? 0);
   const shouldQuarantine = score[test] > QUARANTINE;
   

4. Tag and isolate on threshold crossing. Write the quarantine annotation, open or update a tracking issue, and — critically — make the gate ignore this test’s verdict while keeping its report.

   if (shouldQuarantine && !isQuarantined(test)) {
     await tagQuarantine(test, { score: score[test], reason: 'unstable-selfdiff' });
     await openTrackingIssue(test, score[test]);   // MTTR clock starts here
   }
   

5. Gate on non-quarantined tests only. The merge gate fails on a real, reproducible regression in an active test; a quarantined test can still upload its diff for the dashboard but cannot turn the build red.

   const blocking = results.filter(r => r.verdict === 'fail'
     && !r.unstable && !isQuarantined(r.test));
   process.exit(blocking.length ? 1 : 0);
   

6. Emit metrics for the dashboard. Push flake rate and per-test MTTR so the quarantine population is visible and bounded, not a black hole.

   emitMetric('visual.flake_rate', quarantined.length / totalTests);
   for (const t of quarantined) emitMetric('visual.quarantine_age_days', ageDays(t), { test: t });
   

Cross-browser and cross-environment considerations #

Flake triage changes shape once the same suite runs on more than one engine or runner, because a difference that is a determinism bug in one framing is expected variance in another.

• Score per engine, not per test. A test that is rock-solid on Chromium may be $S=0.3$ on WebKit because that engine lacks the software GL flags that make output deterministic. Track a separate score for test@chromium and test@webkit; a single merged score hides which engine is the problem and quarantines the test everywhere.
• Cross-engine divergence is not flake. If the same commit renders stably on each engine but the engines differ from each other, that is real, reproducible divergence, not non-determinism — the self-diff on each engine is clean. The fix is a per-engine baseline, covered in the Cross-Browser Baseline Matrix, not a quarantine.
• Pin the runner image so the self-diff is trustworthy. The re-run-the-same-commit test only isolates the code when everything else is held: pin the browser image, the font packages, the locale, and --use-gl=swiftshader. If two captures differ because the runner rescheduled onto a node with a different GPU, your self-diff reports a determinism bug that is really an environment leak.
• Watch for shard-dependent flake. A test that is stable alone but flaky when co-located on a busy runner is resource contention — a GPU or CPU starved capture racing the renderer — not a logic bug. Reproduce it under the same parallelism the shard uses before concluding the test itself is at fault.
• Cold versus warm caches. A warm local tile cache can hide the very hydration delay a cold CI runner exposes, so a test that self-diffs clean locally can be unstable in CI. Serve tiles from fixtures and disable HTTP caching so both environments exercise the same timing.

Flake source reference #

Each row maps a common web-map flake source to the signal that identifies it and the fix that removes it. Diagnose against this before loosening a threshold — a looser threshold silences the symptom and hides the next real regression behind it.

Dashboards and metrics #

A quarantine you cannot see becomes a quarantine you never empty. Two metrics keep the loop honest. Flake rate — the share of the active suite currently quarantined — is your gate’s noise floor; a rate creeping up means new flake is entering faster than you clear it, and a rate near zero means the gate is trustworthy. MTTR — the median age of tests in quarantine at the moment they are fixed or deleted — measures whether quarantine is a workshop or a landfill. A rising MTTR with a flat flake rate is the danger signal: tests go in and never come out, coverage quietly erodes, and the dashboard’s green is a lie of omission. Alert on both a flake-rate ceiling (e.g. above 3% of the suite) and a per-test quarantine age ceiling (e.g. any test quarantined longer than 14 days), so a stalled fix escalates instead of decaying into permanent invisibility. Chart the flakiness score distribution too — a broad spread of tests hovering at $S\approx 0.05$ is a slow-burning maintenance cost that a single number hides.

Common pitfalls #

Retries make the suite green but real bugs still ship

Root cause: a retry-until-green policy passes any test that succeeds once out of $k$ attempts, so a partial regression that fails on 40% of runs slips through with a $1-p^k$ probability of a green report — 94% at $k=3$. Diagnose: count how many passing builds required a retry; if a meaningful fraction pass only on attempt 2+, your green is inflated. Fix: cap retries at one or two, record every retry as a flakiness event feeding the score, and quarantine any test that only passes on retry rather than letting the retry silently rescue it.

Quarantine keeps filling up and never empties

Root cause: tests enter quarantine automatically but nothing forces them out, so quarantine becomes a place to hide failures rather than a workshop to fix them. Diagnose: chart MTTR and per-test quarantine age — a rising median with a flat flake rate means tests go in and never leave. Fix: put an age ceiling on quarantine (e.g. 14 days) that escalates a stalled tracking issue, and require every exit to be a recorded human decision to fix or delete — never an automatic re-promotion.

A "flaky" test was actually catching a real intermittent regression

Root cause: the team labelled a test flaky and quarantined it without running the same-commit self-diff, so a genuine partial break — a race in the app itself, not the test — was dismissed as noise. Diagnose: re-run the exact commit twice and diff the two current captures; if they are identical to each other but differ from the baseline, it is reproducible and therefore real, not flake. Fix: make the self-diff mandatory before any quarantine tag, and treat a clean self-diff with a baseline mismatch as a regression to review, never as flake to hide.

The self-diff itself is flaky, so triage is inconclusive

Root cause: the two same-commit captures differ because the runner environment moved between them — a different GPU node, a warm-then-cold cache, an unpinned font — so the self-diff reports non-determinism that is really an environment leak. Diagnose: run both captures on the same pinned image with tiles served from fixtures; if the self-diff now clears, the environment, not the code, was moving. Fix: pin the browser image, fonts, locale, and --use-gl=swiftshader, and serve tiles from fixtures with HTTP caching disabled so both captures see identical inputs.

One noisy engine quarantines a test everywhere

Root cause: a single merged flakiness score across engines crosses the threshold because WebKit is unstable, and the test is pulled from the gate on Chromium too, losing coverage where it was solid. Diagnose: split the score by test@engine and inspect per-engine; a healthy Chromium score next to a poor WebKit one confirms it. Fix: score and quarantine per engine so a flaky engine is isolated without dropping the stable ones, and keep per-engine baselines per the Cross-Browser Baseline Matrix.

Frequently asked questions #

How many reruns do I need to call a map test flaky?

Enough to estimate the rate with confidence, not a single retry. For a quick verdict, re-run the same commit 5–10 times and compute $S=f/N$; a test that reverses even once in ten identical runs is worth watching, and one above roughly $S=0.15$ should be quarantined. For an ongoing signal, maintain an EWMA of the reversal history so you are not re-deciding from scratch every time.

How do I tell a determinism bug from a real regression?

Re-run the exact same commit twice and diff the two current captures against each other, ignoring the baseline. If the two captures differ from each other, the code cannot reproduce its own output, so it is a determinism bug — usually capture racing tile hydration, GPU/font drift, or moving UI. If the two captures are identical to each other but both differ from the baseline, the change is reproducible and therefore a real regression to review.

Does quarantining a test mean I lose its coverage?

No, if you quarantine correctly. A quarantined test still runs, still captures frames, and still reports its diff to the dashboard — it simply cannot fail the merge gate. Coverage is preserved as a visible signal; only the gating veto is removed, and only until the root cause is fixed or the test is deliberately retired. Deleting the test is what loses coverage; quarantine is what avoids having to.

Related #

• CI/CD Integration & Visual Test Operations — the parent area for how map visual tests run, gate, and scale.
• Auto-retrying and quarantining flaky map screenshot tests — the runner configuration that caps retries, tags quarantine, and files the issue.
• Handling Async Tile Loading — the single largest source of capture-race flake, and how to gate it out.
• Noise Reduction for Map Artifacts — absorbing residual GPU and anti-aliasing variance the self-diff surfaces.
• Cross-Browser Baseline Matrix — separating true cross-engine divergence from non-determinism with per-engine baselines.

← Back to CI/CD Integration & Visual Test Operations