Auto-Retrying and Quarantining Flaky Map Screenshot Tests

A map screenshot test fails on the first attempt, passes on the second, and the pipeline goes green. Multiply that by a suite of a few hundred WebGL basemap assertions and the gate stops meaning anything: engineers learn that red is usually noise, click re-run, and eventually merge through a diff that was a genuine regression. The fix is not “add retries” — a blind retry hides regressions just as effectively as it hides flake. This page gives a concrete procedure to retry with attribution, isolate confirmed-nondeterministic tests into a non-blocking quarantine lane that still runs and reports, and prove a failure is flake rather than a regression before you ever quarantine it.

This is one task inside Flaky Visual Test Triage & Quarantine, the guide covering how to classify and contain intermittent map visual failures, and it sits under the broader operations area CI/CD Integration & Visual Test Operations. It assumes you have already removed the common render-timing causes of flake covered in Handling Async Tile Loading — quarantine is for what survives that, not a substitute for it.

Prerequisites

Step-by-step procedure

1. Enable bounded retries and record the attempt count

Set a small, fixed retry ceiling in CI only. Two retries is the working default: enough to let a genuinely nondeterministic test recover, few enough that a hard regression still fails fast instead of burning three runs. Never set retries locally — you want authors to see flake while they write, not have it papered over.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,   // bounded; CI only
  reporter: [
    ['json', { outputFile: 'results/results.json' }],
    ['./reporters/flake-reporter.ts'],  // records attempt outcomes (step 3)
  ],
  projects: [
    { name: 'map-visual', testMatch: /.*\.visual\.spec\.ts/ },
  ],
});

Playwright already exposes the attribution you need: testInfo.retry is the zero-based attempt index, and a test that ends status === 'passed' with retry > 0 is by definition one that only passed on retry. That single fact — passed, but not on the first try — is the raw signal for everything downstream.

2. Tag tests that only pass on retry as flaky

Turn the raw signal into a durable label. In an afterEach hook, detect the pass-on-retry case and annotate the test. The annotation travels into the JSON report, so a later step can read it without re-running anything.

// fixtures/flake-annotate.ts
import { test as base } from '@playwright/test';

export const test = base.extend({});

test.afterEach(async ({}, testInfo) => {
  const passedOnRetry =
    testInfo.status === 'passed' && testInfo.retry > 0;
  if (passedOnRetry) {
    testInfo.annotations.push({
      type: 'flaky',
      description: `passed on attempt ${testInfo.retry + 1}`,
    });
  }
});

A pass-on-retry does not fail the build — Playwright reports the test as flaky, not failed — but the annotation is now a first-class fact you can count, threshold, and act on.

3. Record a flake rate per test

One flaky run is noise; a flake rate is a decision input. Append every attempt outcome to a small rolling history keyed by the test’s stable title, and derive a rate over the last N runs. A custom reporter is the cleanest hook because it sees every result exactly once.

// reporters/flake-reporter.mjs — a custom Playwright reporter
import { readFileSync, writeFileSync, existsSync } from 'node:fs';

const DB = 'results/flake-history.json';

export default class FlakeReporter {
  onTestEnd(test, result) {
    const db = existsSync(DB) ? JSON.parse(readFileSync(DB, 'utf8')) : {};
    const key = test.titlePath().join(' > ');
    const rec = db[key] ?? { runs: 0, flakes: 0 };
    rec.runs += 1;
    if (result.status === 'passed' && result.retry > 0) rec.flakes += 1;
    db[key] = rec;
    writeFileSync(DB, JSON.stringify(db, null, 2));
  }
}

The flake rate for a test is simply

Restore this file from your CI cache at job start and save it at job end so the window spans real history, not a single pipeline. A test whose crosses a policy threshold — 5% over the last 50 runs is a reasonable opening line — is a quarantine candidate, not yet a quarantined test. Confirmation comes next.

4. Confirm nondeterminism with a same-commit double-capture

This is the step that keeps regressions out of quarantine. Before you demote any test, prove its failure is nondeterminism and not a real change: capture the failing viewport twice on the exact same commit, with no code change between shots, and diff the two captures against each other. If the two same-commit captures differ, the test is genuinely nondeterministic and eligible for quarantine. If they are identical to each other but differ from the baseline, the pixels changed deterministically — that is a regression wearing a flake costume, and it must stay in the blocking gate.

// scripts/confirm-flake.spec.ts — run against one suspect test, same commit
import { test, expect } from '@playwright/test';
import { captureMap } from '../fixtures/capture';   // your hardened capture
import pixelmatch from 'pixelmatch';

test('double-capture confirms nondeterminism, not regression', async ({ page }) => {
  const a = await captureMap(page);   // first shot
  const b = await captureMap(page);   // second shot, identical commit
  const selfDiff = pixelmatch(a.data, b.data, null, a.width, a.height,
    { threshold: 0.1 });

  // > 0 differing pixels between two same-commit shots ⇒ nondeterministic.
  // == 0 ⇒ deterministic; any baseline failure is a real regression, DO NOT quarantine.
  console.log(JSON.stringify({ selfDiff }));
  expect(selfDiff, 'stable output — treat baseline diff as a regression')
    .toBeGreaterThan(0);
});

Feed the residual noise this exposes into the filters from Noise Reduction for Map Artifacts before you resort to quarantine — a test whose only nondeterminism is anti-aliasing jitter is better fixed with a comparator tolerance than hidden in a side lane.

5. Quarantine confirmed-flaky tests with a tag script

Quarantine is a @quarantine tag plus a note recording why and when, so the lane never becomes a graveyard of forgotten tests. Apply it with a small script rather than by hand, so the reason and expiry are always attached.

// scripts/quarantine.mjs — usage: node scripts/quarantine.mjs "spec > title" "ticket MVR-812"
import { readFileSync, writeFileSync } from 'node:fs';

const [, , titlePath, reason] = process.argv;
const REG = 'results/quarantine.json';
const reg = JSON.parse(readFileSync(REG, 'utf8'));

reg[titlePath] = {
  reason: reason ?? 'unspecified',
  quarantinedAt: new Date().toISOString().slice(0, 10),
  expiresAt: new Date(Date.now() + 14 * 864e5).toISOString().slice(0, 10),
};
writeFileSync(REG, JSON.stringify(reg, null, 2));
console.log(`Quarantined ${titlePath} until ${reg[titlePath].expiresAt}`);

Read that registry in a global setup and tag matching tests dynamically, or apply test.describe.configure({ tag: '@quarantine' }) at the spec. Every entry carries an expiresAt — the enforceable exit criterion from step 7.

6. Route quarantined tests to a non-blocking job that still runs

The critical rule: quarantined tests keep running and keep reporting — they simply cannot fail the merge. Split the workflow into a blocking job that runs everything except @quarantine, and a second job that runs only @quarantine with continue-on-error. The lane stays visible (you will see it flip red/green), so a quarantined test that starts passing consistently, or one that begins failing its own same-commit double-capture, surfaces instead of rotting silently.

# .github/workflows/map-visual.yml
jobs:
  visual-gate:                       # blocking — merges depend on this
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright test --grep-invert @quarantine

  visual-quarantine:                 # non-blocking, but still executes + reports
    runs-on: ubuntu-latest
    continue-on-error: true          # never blocks the merge
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright test --grep @quarantine
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: quarantine-report, path: results/ }

Wire this alongside your primary thresholds as described in GitHub Actions & GitLab CI Gates for Map Visual Tests; only the blocking visual-gate job should be a required status check.

7. Set an exit criterion: fix or delete

Quarantine is a holding pattern, not a destination. Enforce the expiresAt from step 5 with a job that fails the build when a quarantined entry is past due, forcing a decision — fix the root cause and un-quarantine, or delete the test as not worth its cost. A test that cannot be made deterministic and is not worth deleting is telling you the assertion is wrong.

// scripts/enforce-expiry.mjs — run in CI; nonzero exit fails the build
import { readFileSync } from 'node:fs';
const reg = JSON.parse(readFileSync('results/quarantine.json', 'utf8'));
const today = new Date().toISOString().slice(0, 10);
const overdue = Object.entries(reg).filter(([, v]) => v.expiresAt < today);
if (overdue.length) {
  console.error('Quarantine expired — fix or delete:',
    overdue.map(([k]) => k));
  process.exit(1);
}
Decision flow from a failing map screenshot test to the quarantine lane or the blocking gate A failing test on its first attempt enters bounded retry. If it passes on retry it is tagged flaky and a same-commit double-capture is run. If the two same-commit captures differ from each other, the failure is confirmed nondeterministic and the test is routed to a non-blocking quarantine lane that still runs and reports, with a fix-or-delete expiry. If the two same-commit captures are identical to each other, the change is deterministic and the test is routed to the blocking regression gate, which fails the merge. A test that fails every retry also goes straight to the blocking gate. Test fails first attempt Bounded retry pass on retry? Same-commit double-capture two shots differ? Quarantine lane non-blocking · still runs + reports flake rate tracked · fix-or-delete expiry Blocking regression gate deterministic change · fails the merge real regression, not flake yes · flaky no · stable fails every retry

Verification

Troubleshooting

Symptom Likely cause Fix
A real regression landed in quarantine and shipped Test was demoted on a raw retry signal without the same-commit double-capture, so a deterministic change was misread as flake Make step 4 mandatory before any quarantine; a test whose two same-commit captures match must stay in the blocking gate
Quarantine lane grows every sprint and nothing leaves No enforced exit criterion, so demotion is one-way Attach expiresAt on quarantine (step 5) and fail the build on overdue entries (step 7) — fix the root cause or delete
Flake rate reads 0% for a known-flaky test History file not restored between pipelines, so every run starts fresh Cache and restore flake-history.json at job start and save at job end so the window spans real history

Frequently asked questions

Doesn't adding retries just hide real regressions?

A blind retry does. This procedure retries with attribution: a test that only passes on retry is tagged flaky, not treated as a clean pass, and it is never quarantined until a same-commit double-capture proves its output is genuinely nondeterministic. A deterministic pixel change fails that check and stays in the blocking gate, so a regression cannot be retried into a merge.

Why run quarantined tests at all if they can't fail the build?

Because a quarantined test that has been fixed, or one that has quietly turned into a real regression, only shows up if it keeps running. Routing quarantine to a non-blocking job that still executes and reports keeps the signal visible: you watch the lane flip green (ready to un-quarantine) or start failing its own double-capture (now a regression), instead of the test rotting silently behind a skip.

What flake rate should trigger quarantine?

Start at 5% over a rolling window of about 50 runs, then tune to your suite’s tolerance. The rate is a candidate filter, not the decision — it tells you which tests are worth the same-commit double-capture in step 4. Confirmation of nondeterminism, not the rate alone, is what authorizes quarantine.

How is this different from just skipping the flaky test?

A skip stops running the test, so you lose all signal — you cannot tell whether it was fixed or silently became a regression, and it never expires. Quarantine keeps the test running in a non-blocking lane, tracks its flake rate, and carries a fix-or-delete deadline, so it is a temporary, observable holding pattern rather than a permanent blind spot.

← Back to CI/CD Integration & Visual Test Operations