Auto-Retrying and Quarantining Flaky Map Screenshot Tests

A map screenshot test fails on the first attempt, passes on the second, and the pipeline goes green. Multiply that by a suite of a few hundred WebGL basemap assertions and the gate stops meaning anything: engineers learn that red is usually noise, click re-run, and eventually merge through a diff that was a genuine regression. The fix is not “add retries” — a blind retry hides regressions just as effectively as it hides flake. This page gives a concrete procedure to retry with attribution, isolate confirmed-nondeterministic tests into a non-blocking quarantine lane that still runs and reports, and prove a failure is flake rather than a regression before you ever quarantine it.

This is one task inside Flaky Visual Test Triage & Quarantine, the guide covering how to classify and contain intermittent map visual failures, and it sits under the broader operations area CI/CD Integration & Visual Test Operations. It assumes you have already removed the common render-timing causes of flake covered in Handling Async Tile Loading — quarantine is for what survives that, not a substitute for it.

Prerequisites

Step-by-step procedure

1. Enable bounded retries and record the attempt count

Set a small, fixed retry ceiling in CI only. Two retries is the working default: enough to let a genuinely nondeterministic test recover, few enough that a hard regression still fails fast instead of burning three runs. Never set retries locally — you want authors to see flake while they write, not have it papered over.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,   // bounded; CI only
  reporter: [
    ['json', { outputFile: 'results/results.json' }],
    ['./reporters/flake-reporter.ts'],  // records attempt outcomes (step 3)
  ],
  projects: [
    { name: 'map-visual', testMatch: /.*\.visual\.spec\.ts/ },
  ],
});

Playwright already exposes the attribution you need: testInfo.retry is the zero-based attempt index, and a test that ends status === 'passed' with retry > 0 is by definition one that only passed on retry. That single fact — passed, but not on the first try — is the raw signal for everything downstream.

2. Tag tests that only pass on retry as flaky

Turn the raw signal into a durable label. In an afterEach hook, detect the pass-on-retry case and annotate the test. The annotation travels into the JSON report, so a later step can read it without re-running anything.

// fixtures/flake-annotate.ts
import { test as base } from '@playwright/test';

export const test = base.extend({});

test.afterEach(async ({}, testInfo) => {
  const passedOnRetry =
    testInfo.status === 'passed' && testInfo.retry > 0;
  if (passedOnRetry) {
    testInfo.annotations.push({
      type: 'flaky',
      description: `passed on attempt ${testInfo.retry + 1}`,
    });
  }
});

A pass-on-retry does not fail the build — Playwright reports the test as flaky, not failed — but the annotation is now a first-class fact you can count, threshold, and act on.

3. Record a flake rate per test

One flaky run is noise; a flake rate is a decision input. Append every attempt outcome to a small rolling history keyed by the test’s stable title, and derive a rate over the last N runs. A custom reporter is the cleanest hook because it sees every result exactly once.

// reporters/flake-reporter.mjs — a custom Playwright reporter
import { readFileSync, writeFileSync, existsSync } from 'node:fs';

const DB = 'results/flake-history.json';

export default class FlakeReporter {
  onTestEnd(test, result) {
    const db = existsSync(DB) ? JSON.parse(readFileSync(DB, 'utf8')) : {};
    const key = test.titlePath().join(' > ');
    const rec = db[key] ?? { runs: 0, flakes: 0 };
    rec.runs += 1;
    if (result.status === 'passed' && result.retry > 0) rec.flakes += 1;
    db[key] = rec;
    writeFileSync(DB, JSON.stringify(db, null, 2));
  }
}

The flake rate for a test is simply

Restore this file from your CI cache at job start and save it at job end so the window spans real history, not a single pipeline. A test whose crosses a policy threshold — 5% over the last 50 runs is a reasonable opening line — is a quarantine candidate, not yet a quarantined test. Confirmation comes next.

The flake rate is a rolling window, not a running total Two ways of counting the same test's history are compared over thirty runs. A lifetime total mixes an old, already-fixed instability with recent behaviour and keeps the test above the quarantine threshold long after it stopped flaking. A rolling window of the last twenty runs drops the fixed period out of the calculation and shows the rate falling below the threshold, which is what releases the test from quarantine. A threshold line at two percent crosses both series. A note gives the practical rule: a window short enough to react and long enough that one bad run cannot cross the line on its own. A rolling window releases a fixed test; a lifetime total never does quarantine threshold 2% lifetime total rolling 20 runs instability fixed here flake % short enough to react, long enough that a single bad run cannot cross the line on its own twenty runs is a reasonable default for a suite that runs on every pull request

4. Confirm nondeterminism with a same-commit double-capture

This is the step that keeps regressions out of quarantine. Before you demote any test, prove its failure is nondeterminism and not a real change: capture the failing viewport twice on the exact same commit, with no code change between shots, and diff the two captures against each other. If the two same-commit captures differ, the test is genuinely nondeterministic and eligible for quarantine. If they are identical to each other but differ from the baseline, the pixels changed deterministically — that is a regression wearing a flake costume, and it must stay in the blocking gate.

// scripts/confirm-flake.spec.ts — run against one suspect test, same commit
import { test, expect } from '@playwright/test';
import { captureMap } from '../fixtures/capture';   // your hardened capture
import pixelmatch from 'pixelmatch';

test('double-capture confirms nondeterminism, not regression', async ({ page }) => {
  const a = await captureMap(page);   // first shot
  const b = await captureMap(page);   // second shot, identical commit
  const selfDiff = pixelmatch(a.data, b.data, null, a.width, a.height,
    { threshold: 0.1 });

  // > 0 differing pixels between two same-commit shots ⇒ nondeterministic.
  // == 0 ⇒ deterministic; any baseline failure is a real regression, DO NOT quarantine.
  console.log(JSON.stringify({ selfDiff }));
  expect(selfDiff, 'stable output — treat baseline diff as a regression')
    .toBeGreaterThan(0);
});

Feed the residual noise this exposes into the filters from Noise Reduction for Map Artifacts before you resort to quarantine — a test whose only nondeterminism is anti-aliasing jitter is better fixed with a comparator tolerance than hidden in a side lane.

5. Quarantine confirmed-flaky tests with a tag script

Quarantine is a @quarantine tag plus a note recording why and when, so the lane never becomes a graveyard of forgotten tests. Apply it with a small script rather than by hand, so the reason and expiry are always attached.

// scripts/quarantine.mjs — usage: node scripts/quarantine.mjs "spec > title" "ticket MVR-812"
import { readFileSync, writeFileSync } from 'node:fs';

const [, , titlePath, reason] = process.argv;
const REG = 'results/quarantine.json';
const reg = JSON.parse(readFileSync(REG, 'utf8'));

reg[titlePath] = {
  reason: reason ?? 'unspecified',
  quarantinedAt: new Date().toISOString().slice(0, 10),
  expiresAt: new Date(Date.now() + 14 * 864e5).toISOString().slice(0, 10),
};
writeFileSync(REG, JSON.stringify(reg, null, 2));
console.log(`Quarantined ${titlePath} until ${reg[titlePath].expiresAt}`);

Read that registry in a global setup and tag matching tests dynamically, or apply test.describe.configure({ tag: '@quarantine' }) at the spec. Every entry carries an expiresAt — the enforceable exit criterion from step 7.

6. Route quarantined tests to a non-blocking job that still runs

The critical rule: quarantined tests keep running and keep reporting — they simply cannot fail the merge. Split the workflow into a blocking job that runs everything except @quarantine, and a second job that runs only @quarantine with continue-on-error. The lane stays visible (you will see it flip red/green), so a quarantined test that starts passing consistently, or one that begins failing its own same-commit double-capture, surfaces instead of rotting silently.

# .github/workflows/map-visual.yml
jobs:
  visual-gate:                       # blocking — merges depend on this
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright test --grep-invert @quarantine

  visual-quarantine:                 # non-blocking, but still executes + reports
    runs-on: ubuntu-latest
    continue-on-error: true          # never blocks the merge
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright test --grep @quarantine
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: quarantine-report, path: results/ }

Wire this alongside your primary thresholds as described in GitHub Actions & GitLab CI Gates for Map Visual Tests; only the blocking visual-gate job should be a required status check.

7. Set an exit criterion: fix or delete

Quarantine is a holding pattern, not a destination. Enforce the expiresAt from step 5 with a job that fails the build when a quarantined entry is past due, forcing a decision — fix the root cause and un-quarantine, or delete the test as not worth its cost. A test that cannot be made deterministic and is not worth deleting is telling you the assertion is wrong.

// scripts/enforce-expiry.mjs — run in CI; nonzero exit fails the build
import { readFileSync } from 'node:fs';
const reg = JSON.parse(readFileSync('results/quarantine.json', 'utf8'));
const today = new Date().toISOString().slice(0, 10);
const overdue = Object.entries(reg).filter(([, v]) => v.expiresAt < today);
if (overdue.length) {
  console.error('Quarantine expired — fix or delete:',
    overdue.map(([k]) => k));
  process.exit(1);
}
Decision flow from a failing map screenshot test to the quarantine lane or the blocking gate A failing test on its first attempt enters bounded retry. If it passes on retry it is tagged flaky and a same-commit double-capture is run. If the two same-commit captures differ from each other, the failure is confirmed nondeterministic and the test is routed to a non-blocking quarantine lane that still runs and reports, with a fix-or-delete expiry. If the two same-commit captures are identical to each other, the change is deterministic and the test is routed to the blocking regression gate, which fails the merge. A test that fails every retry also goes straight to the blocking gate. Test fails first attempt Bounded retry pass on retry? Same-commit double-capture two shots differ? Quarantine lane non-blocking · still runs + reports flake rate tracked · fix-or-delete expiry Blocking regression gate deterministic change · fails the merge real regression, not flake flaky stable fails every retry Quarantine with an expiry date, and the two ways out of it A quarantined test enters a timed lane with an explicit expiry. While inside, it keeps running and keeps reporting, but cannot block a merge. Two exits are drawn. The first is a fix: the flake rate falls below the threshold across a full window, and the test returns to the blocking suite. The second is the expiry: the date passes with no fix, and the test is deleted rather than left running. A footer explains why deletion is the right second exit — a quarantined test that nobody fixes is a test nobody trusts, and keeping it costs runner minutes while giving no signal. Quarantine is a lane with a deadline, not a parking space quarantine lane still runs, still reports, cannot block a merge fixed expired back to the blocking suite flake rate under the threshold across a full window deleted a test nobody fixes is a test nobody trusts — it only costs minutes

Verification

Troubleshooting

Symptom Likely cause Fix
A real regression landed in quarantine and shipped Test was demoted on a raw retry signal without the same-commit double-capture, so a deterministic change was misread as flake Make step 4 mandatory before any quarantine; a test whose two same-commit captures match must stay in the blocking gate
Quarantine lane grows every sprint and nothing leaves No enforced exit criterion, so demotion is one-way Attach expiresAt on quarantine (step 5) and fail the build on overdue entries (step 7) — fix the root cause or delete
Flake rate reads 0% for a known-flaky test History file not restored between pipelines, so every run starts fresh Cache and restore flake-history.json at job start and save at job end so the window spans real history

Frequently asked questions

Doesn't adding retries just hide real regressions?

A blind retry does. This procedure retries with attribution: a test that only passes on retry is tagged flaky, not treated as a clean pass, and it is never quarantined until a same-commit double-capture proves its output is genuinely nondeterministic. A deterministic pixel change fails that check and stays in the blocking gate, so a regression cannot be retried into a merge.

Why run quarantined tests at all if they can't fail the build?

Because a quarantined test that has been fixed, or one that has quietly turned into a real regression, only shows up if it keeps running. Routing quarantine to a non-blocking job that still executes and reports keeps the signal visible: you watch the lane flip green (ready to un-quarantine) or start failing its own double-capture (now a regression), instead of the test rotting silently behind a skip.

What flake rate should trigger quarantine?

Start at 5% over a rolling window of about 50 runs, then tune to your suite’s tolerance. The rate is a candidate filter, not the decision — it tells you which tests are worth the same-commit double-capture in step 4. Confirmation of nondeterminism, not the rate alone, is what authorizes quarantine.

How is this different from just skipping the flaky test?

A skip stops running the test, so you lose all signal — you cannot tell whether it was fixed or silently became a regression, and it never expires. Quarantine keeps the test running in a non-blocking lane, tracks its flake rate, and carries a fix-or-delete deadline, so it is a temporary, observable holding pattern rather than a permanent blind spot.

← Back to CI/CD Integration & Visual Test Operations