Testing Map Tiles After CDN Cache Invalidation

You ship a restyle — a new road casing colour, a shifted label font, a corrected polygon fill — and push a purge to the tile CDN. The visual test runs green. Weeks later a user reports the old style is still on the map. The test never lied about the pixels it saw; it saw pixels served from a warm edge cache that the purge never reached, and it happily matched them against a baseline captured from the same stale cache. A CDN sits between your origin tiles and the browser precisely to avoid cold fetches, which means the default path through your test suite is the one path that cannot prove invalidation worked. This page gives a concrete procedure to force a cold fetch, compare warm against fresh, and assert that the new tiles are actually visible before you trust the gate.

This is one task inside Cache & CDN Invalidation Testing, the guide that treats the cache layer as a source of UI instability. It sits under the broader area on Dynamic Element Masking & UI Stability, and it assumes you already gate capture on full tile hydration per Handling Async Tile Loading — a stale tile that paints instantly is still a fully hydrated frame, so hydration gating alone will never catch this class of bug.

Prerequisites

Step-by-step procedure

1. Identify the cache key that actually varies

A CDN caches a tile against a key, and if your key does not change when the tile content changes, no purge will ever be observable. For a {z}/{x}/{y} tile the key is the request URL plus whatever headers the edge is configured to vary on. The most robust pattern is a version segment or query parameter that moves with every style or data release — /{styleVersion}/{z}/{x}/{y}.png or ?v=<styleHash> — so a new release fetches a new key rather than relying on a purge of the old one. Capture the key and the freshness headers before you change anything, so you have a baseline to compare against.

const probe = await page.request.get(
  'https://tiles.example.com/v3/12/654/1583.png'
);
console.log({
  cacheControl: probe.headers()['cache-control'],   // e.g. "public, max-age=86400"
  etag: probe.headers()['etag'],                     // strong validator, if present
  age: probe.headers()['age'],                       // seconds this object has lived at the edge
  status: probe.headers()['x-cache'],                // "Hit from cloudfront" / "MISS"
});

If Cache-Control carries a long max-age and no version segment appears in the path, the key never rotates — fix that first, because the rest of the procedure only measures a cache you can move.

2. Trigger the invalidation and wait for it to settle

A purge is asynchronous. CloudFront reports an invalidation as InProgress before Completed; Fastly and Cloudflare propagate across POPs over seconds. Firing the purge and immediately screenshotting races the propagation and gives you a false result in either direction. Trigger the invalidation, then poll the provider until it reports done — do not sleep on a fixed timer.

# CloudFront: create the invalidation and block until it completes
ID=$(aws cloudfront create-invalidation \
  --distribution-id "$DIST_ID" \
  --paths "/v3/12/654/1583.png" \
  --query 'Invalidation.Id' --output text)

aws cloudfront wait invalidation-completed \
  --distribution-id "$DIST_ID" --id "$ID"

If you rotate the cache key instead of purging (the safer strategy), there is nothing to wait for — the new key is cold by definition, and you can skip straight to a cold fetch of the new URL.

3. Force a cold fetch inside the test

This is the load-bearing step. Left alone, the browser and the edge will both hand you the warmest copy they have. To prove the origin serves the new tile, defeat both caches for the assertion request: send Cache-Control: no-cache from the client so the edge revalidates, and append a unique cache-busting parameter so no intermediate layer can key onto a stale object. Confirm the response came back a MISS — a Hit here means you are still measuring the old cache.

const bust = `cb=${Date.now()}`;
await page.route('**/v3/**/*.png', async (route) => {
  const url = new URL(route.request().url());
  url.searchParams.set('cb', bust);              // unique key defeats edge + browser cache
  const response = await route.fetch({
    url: url.toString(),
    headers: { ...route.request().headers(), 'cache-control': 'no-cache' },
  });
  const edgeStatus = response.headers()['x-cache'] || response.headers()['cf-cache-status'];
  if (/hit/i.test(edgeStatus || '')) {
    throw new Error(`Expected a cold fetch, got edge ${edgeStatus} for ${url.pathname}`);
  }
  await route.fulfill({ response });
});

4. Capture warm and fresh renders as two distinct frames

Now render the map twice against the same locked camera: once through the ordinary warm path (no cache-busting, exactly what a real user hits) and once through the cold path from step 3. Gate both captures on full tile hydration so neither frame is half-painted — the warm/fresh distinction is meaningless if either capture raced the renderer, which is the whole subject of Handling Async Tile Loading.

async function renderLockedFrame(page, { cold }) {
  await page.evaluate(() => window.__cacheBust = false);
  if (cold) await page.evaluate(() => window.__cacheBust = true);
  await page.goto('http://localhost:8080/map.html?z=12&x=654&y=1583');
  await page.evaluate(() => new Promise((resolve) => {
    const map = window.mapInstance;
    const settled = () => !map.isMoving() && map.areTilesLoaded();
    const check = () => settled() && (map.off('idle', check), resolve());
    map.on('idle', check); check();
  }));
  return page.locator('#map').screenshot({ animations: 'disabled' });
}

const warm = await renderLockedFrame(page, { cold: false });
const fresh = await renderLockedFrame(page, { cold: true });

5. Assert the fresh frame differs from the pre-purge baseline

Invalidation is proven not by warm and fresh matching, but by the fresh frame matching the new expected tiles and diverging from the pre-purge baseline. Diff the fresh render against the baseline you captured before the release: a genuine invalidation makes them differ in exactly the region that changed. Use a structural comparison so anti-aliasing noise does not masquerade as the change you are hunting.

import pixelmatch from 'pixelmatch';
import { PNG } from 'pngjs';

const oldBaseline = PNG.sync.read(fs.readFileSync('baseline/pre-purge-z12.png'));
const freshPng = PNG.sync.read(fresh);
const { width, height } = oldBaseline;
const changed = pixelmatch(
  oldBaseline.data, freshPng.data, null, width, height,
  { threshold: 0.1, includeAA: false }
);
const ratio = changed / (width * height);

// The restyle MUST be visible: fresh tiles differ from the old baseline.
if (ratio < 0.002) {
  throw new Error(
    `Stale cache suspected: fresh render is ${(ratio * 100).toFixed(3)}% ` +
    `different from the pre-purge baseline — the new tiles are not being served.`
  );
}

The polarity of this assertion is the point. A conventional visual test fails when things change; this one fails when nothing changed, because after a deliberate restyle no change is the regression.

6. Re-bless the baseline to the new style key

Once you have confirmed the fresh tiles are correct, promote the fresh render to the approved baseline and tag it against the new style version — not the purge timestamp. Keying the baseline to the same version segment that rotates the cache means the golden and the cache move together, so the next release cannot silently reuse an old golden. This is the versioning discipline covered in Baseline Management for Tile Servers; apply it here so the cache key and the baseline share one source of truth.

cp current/fresh-z12.png baselines/v3/home-z12.png
# manifest records the key the cache and the baseline both derive from
jq '.styleVersion = "v3" | .capturedAt = now' \
  baselines/v3/home-z12.json > tmp && mv tmp baselines/v3/home-z12.json
Warm and cold request paths across the CDN invalidation boundary A tile request from the test browser reaches the CDN edge cache. On the warm path the edge returns a Hit — the pre-purge tile served straight from cache without touching the origin, which is how a visual test silently passes on stale tiles. An invalidation boundary marks a purge or cache-key rotation. On the cold path, forced by a cache-busting parameter and a no-cache header, the edge reports a Miss and fetches the new tile from the origin tile server, returning the fresh restyled tile. The test asserts the fresh frame differs from the pre-purge baseline to prove invalidation reached the origin. Hit stays stale · Miss proves the purge reached origin Test browser requests {z}/{x}/{y} CDN edge cache keyed on URL + vary X-Cache: Hit / Miss Origin tiles fresh restyled tile warm: Hit → stale tile cold: Miss → origin cb param + no-cache invalidation boundary — purge / key rotation

Verification

Confirm the check actually distinguishes fresh from stale before you rely on it:

Troubleshooting

Symptom Likely cause Fix
Test passes but users still see the old style Cold-fetch step skipped, so the assertion read the warm edge cache Force the cache-busting param and no-cache header from step 3 and assert an edge miss before capturing
Cold fetch still returns a Hit Edge varies on a header the buster does not touch, or the param is stripped upstream Rotate the version segment in the path itself rather than a query param, so the cache key changes wholesale
Assertion flakes right after purge Screenshot raced propagation across POPs before the invalidation completed Poll the provider to Completed in step 2 instead of a fixed sleep, then fetch

Frequently asked questions

Why does my visual test pass while users still see the old tiles?

Because the test read the same warm edge cache the purge failed to clear, and matched it against a baseline captured from that same stale cache — a self-consistent false pass. Force a cold fetch for the assertion with a cache-busting parameter and a Cache-Control: no-cache header, confirm the edge reports a miss, and assert the fresh frame differs from the pre-purge baseline.

Should I purge the cache or rotate the cache key?

Rotate the key when you can. A version segment like /v3/{z}/{x}/{y}.png makes every release fetch a cold key by definition, so there is no asynchronous purge to wait on and no window where some POPs serve old tiles and others new. Reserve explicit purges for correcting bad tiles under a key you cannot rotate.

Why assert that the fresh frame *differs* from the baseline?

After a deliberate restyle, sameness is the failure. A normal visual test fails on change; this one inverts the polarity and fails when the fresh render matches the pre-purge baseline, because that match means the new tiles never reached the browser. Once you confirm the difference is correct, re-bless the baseline to the new style version.

Do I still need this if I already gate on tile hydration?

Yes. Hydration gating from Handling Async Tile Loading guarantees the frame is fully painted, but a stale cached tile paints just as completely as a fresh one. Hydration proves the frame finished; cache-invalidation testing proves the frame is made of the right tiles.