Map Library Version Upgrade Testing

A map library upgrade is the one change guaranteed to turn a visual suite red across the board, and it is also the change the suite is most valuable for. Every other kind of regression is local; a renderer version bump can move every glyph, every anti-aliased edge and every collision decision at once. The question is never “did anything change” — something always did — but “is everything that changed something we accept”. This page is about running an upgrade so that question has an answer, and so the resulting baseline churn stays attributable a year later.

This page sits under Web Map Visual Testing Fundamentals & Toolchains and leans on two others: the storage and keying model in Baseline Management for Tile Servers, and the per-engine reasoning in Cross-Browser Baseline Matrix — a library version behaves like an engine axis, and for the same reasons.

Three kinds of change a version bump produces

A library upgrade mixes three categories in one diff, and almost all of the difficulty comes from not separating them.

Deliberate rendering changes. The release notes say symbol placement was improved, or the text halo is now rendered differently, or a shader was rewritten. These are the library working as intended, and the correct response is to accept them — but as an attributed batch, not silently.

Incidental rendering changes. Nothing in the release notes mentions it, but a rounding change in the tile-coordinate maths moved every road casing by a subpixel. This is also the library working, in the sense that nothing is broken, and it is exactly what a visual suite is for: nobody would have found it any other way.

Regressions. A layer type stopped honouring a paint property, a projection edge case broke at high latitude, a minzoom is now applied inclusively where it was exclusive. These are the reason the upgrade is being tested, and they are drowned by the first two categories unless the review separates them.

The separating tool is the grouping described in Baseline Review & Approval Workflows: a frame-wide uniform change groups into one row, and the single scenario whose signature matches nothing is where the regression is.

One upgrade diff, three categories, three different responses The failures from a single version bump are split into three bands sized by how many captures fall into each. The largest band is deliberate rendering changes, described in the release notes, whose response is to accept them as one attributed batch. The middle band is incidental rendering changes that no release note mentions, whose response is also acceptance but with the difference recorded, because nothing else would have found them. The smallest band is regressions, whose response is to stop the upgrade and report upstream. A caption notes that the smallest band is the reason for the exercise and is invisible unless the other two are collapsed first. The band that matters is the one the other two hide deliberate · in the release notes accept as one attributed batch — the annotation names the version, not the branch incidental · nothing mentions it accept and record — nothing else would have found it regression stop, report ← the reason for the exercise, invisible until the two above collapse into rows

Design: treat the library version as a baseline axis

The mistake that makes upgrades painful is treating the library version as ambient — something the lockfile records and the baseline key ignores. Under that model an upgrade rewrites every baseline in place, the old ones are gone, and there is no way to compare the two versions after the fact.

Putting the major version into the key, as Baseline Management for Tile Servers already does for the engine, changes the shape of the work entirely:

baselines/downtown-z14/maplibre-4/chromium-121/dpr1/style-a4f9.png
baselines/downtown-z14/maplibre-5/chromium-121/dpr1/style-a4f9.png

Now the upgrade branch adds a cell rather than overwriting one. Three things become possible that were not before. Both versions can be captured in the same run, so the comparison is between two frames from one machine at one instant rather than between today and whenever the old baseline was blessed. The old version keeps working while the new one is evaluated, so an upgrade can be abandoned without a restore. And the two cells can be diffed directly, which is the measurement the decision actually needs.

Whether to keep both cells after the upgrade lands is a retention question rather than a correctness one. Keeping the outgoing version’s cell for one release cycle costs little and makes “did this look different before the upgrade” answerable without archaeology.

Step-by-step: running the upgrade

1. Freeze everything except the library. The upgrade branch changes one dependency and nothing else — not the browser, not the fonts, not the style, not the fixtures. Any other change in the same branch makes the diff uninterpretable, and the temptation to bundle a browser bump with a library bump is strong precisely because both cause churn.

2. Capture both versions in one run. A matrix job with the library version as the axis, both cells writing to their own key.

strategy:
  matrix:
    maplibre: ['4.7.1', '5.0.0']
steps:
  - run: npm install maplibre-gl@$ --no-save
  - run: npm run capture -- --key-version=maplibre-$

3. Diff the two cells against each other, not against history. This is the measurement. Comparing the new version against the stored baseline conflates the library change with everything that has drifted since that baseline was blessed; comparing the two cells from the same run isolates the library.

4. Group and triage. Frame-wide uniform changes are one row. Read the release notes against the row descriptions and mark each group deliberate or incidental. Anything that does not group is a candidate regression and gets looked at individually.

5. Reproduce each candidate regression outside the suite. A minimal page with the smallest style that shows it. This is what makes an upstream report actionable, and it is also how a suite bug — a fixture that assumed an internal detail, a readiness predicate that used a private method — gets found before it is reported as a library defect.

6. Bless the new cell as one attributed batch. The annotation records the version pair, the release, the reviewer and the group signatures accepted, per the audit fields in Baseline Review & Approval Workflows. This is what makes the churn attributable in a year.

7. Switch the default and keep the old cell for one cycle. The suite now compares against the new version by default; the old cell remains readable, which costs a little storage and removes the need for a restore if the upgrade has to be reverted.

Diffing the two version cells against each other rather than against stored history Two comparison shapes are contrasted. In the first, the new version's capture is diffed against the stored baseline, and an arrow shows that the stored baseline was blessed months earlier — so the measured difference contains the library change plus every environment and style drift accumulated since. In the second, both versions are captured in the same run on the same machine at the same instant and diffed against each other, so the only variable is the library. A caption states the consequence: only the second measurement can be reported upstream, because only it isolates the change being evaluated. Compare the two versions, not the new one against history stored baseline blessed in March new version, today new browser, new fonts too library change + 5 months of drift not reportable, not decidable old version, today same run, same machine new version, today same run, same machine library change only reportable upstream as-is the matrix job that captures both cells costs one extra run and makes the difference measurable

What differs between the major libraries

The procedure is common; where the churn comes from is not.

  • MapLibre GL JS and Mapbox GL JS upgrades move pixels most, because the renderer is doing the drawing. Symbol placement, text halo rendering, line joins and the fade timings between tile levels all change between majors, and a minor can move anti-aliasing. Expect frame-wide churn and budget for a full re-bless.
  • Leaflet upgrades move pixels least, because the browser draws the tiles and Leaflet positions them. The churn that does appear is usually in control chrome, marker anchoring and the exact transform used for panning — localised, easy to group, and easy to review. Behavioural changes to event handling matter more than rendering ones.
  • OpenLayers sits between the two: it renders vector data itself but composes raster tiles through the DOM or a canvas depending on the layer type, so an upgrade can produce frame-wide churn on the vector layers and none on the raster ones in the same capture. Grouping by region class rather than by frame is what makes that legible.
  • deck.gl and other overlay libraries are separate axes again. An overlay library upgrade changes only its own layer, which makes it the easiest case to evaluate and the easiest to miss if the overlay is one of the regions the suite masks.

Behavioural changes the pixels will not show

A visual suite is a strong instrument for a library upgrade and a partial one, and the gap is worth naming before an upgrade is signed off on its evidence alone.

Event semantics. A release that changes when moveend fires relative to idle, or that stops emitting an event for a programmatic camera change, produces byte-identical captures and breaks every harness built on the old ordering. Ironically the visual suite is usually the first thing to notice, not through a diff but through a timeout: captures start hanging because the readiness predicate is waiting for an event that no longer arrives. Treat a suite-wide timeout during an upgrade as a signal about event semantics rather than as an infrastructure problem.

Interaction handling. Inertia curves, wheel-zoom sensitivity, touch gesture thresholds and the exact camera a double-click lands on all change between versions, and none of them appear in a capture taken from a locked camera with handlers disabled. These belong to a functional suite, and an upgrade evaluation should say explicitly whether that suite ran.

Memory and resource behaviour. A version that holds more tiles in cache, or that no longer releases WebGL resources on remove(), produces identical frames and a different failure profile in production. The visual suite may notice indirectly — runs getting slower, or a long matrix job hitting a memory limit that it did not before — and that observation is worth recording rather than working around with a larger runner.

API deprecations that still work. A method that logs a warning today and is removed in the next major leaves no trace in a capture. Running the upgrade branch with console warnings promoted to failures is a cheap addition to the evaluation and catches the deprecations while there is still a version where both the old and new call sites work.

Recording which of these were checked, alongside the visual result, is what turns “the baselines were re-blessed” into a decision someone can stand behind. The visual evidence answers whether the map still looks right; these four answer whether it still behaves right, and an upgrade needs both.

Upgrade parameter reference

Decision Recommended default Notes
Version segment in the baseline key major only Minors rarely move pixels enough to justify a cell; when one does, pin the exact version in that cell’s annotation
Cells captured per upgrade run both, in one matrix The whole point is comparing them under identical conditions
Old cell retention after the switch one release cycle Cheap, and removes the need for a restore on a revert
Other changes allowed in the branch none A bundled browser bump makes the diff uninterpretable
Scenarios captured for the evaluation the full matrix An upgrade is exactly the case where a sampled run is not enough
Grouping bucket width wider than usual Frame-wide churn should collapse into a handful of rows, not hundreds
Regression reproduction minimal page, outside the suite The form an upstream report needs, and how suite bugs get caught
What a capture proves about an upgrade, and what it cannot Two columns divide an upgrade's risk surface. The left column lists what a visual capture answers directly: symbol placement, glyph rendering, line joins and caps, colour and ramp evaluation, collision and label thinning, and tile fade behaviour. The right column lists what produces byte-identical captures and still breaks: event ordering and emission, interaction curves and gesture thresholds, memory and resource release, and deprecated methods that still work today. A note under the right column names the cheap checks that cover it — a functional suite run, console warnings promoted to failures, and watching for suite-wide timeouts during the upgrade. Sign the upgrade off on both columns, not just the left one the capture answers this symbol placement glyph rendering and halos line joins, caps, casings colour and ramp evaluation collision and label thinning tile fade behaviour identical pixels, still broken event ordering and emission interaction curves, gestures memory and resource release deprecations that still work covered by: functional suite, warnings-as-errors, and treating a suite-wide timeout as a signal record which right-column checks ran, next to the visual result — that is the sign-off

Common pitfalls

Bundling the upgrade with anything else. The single most common cause of an upgrade that takes three weeks. Every additional change in the branch multiplies the interpretations of every group in the diff.

Re-blessing before triaging. Accepting the new captures to make the branch green, intending to look properly afterwards, destroys the comparison — the old cell is gone and the regressions are now the reference. The batch approval is the last step for a reason.

Sampling the suite for the evaluation. Running a subset because the full matrix is slow is reasonable on a feature branch and wrong here: the regressions an upgrade introduces are concentrated in unusual scenarios, which is exactly what a sample drops.

Assuming a minor cannot move pixels. Renderer minors regularly include shader and placement fixes. The key does not need a cell per minor, but the upgrade still needs the two-cell comparison; the difference is only whether the result is stored separately.

Reporting a diff as an upstream bug without a minimal reproduction. A screenshot of an application with a proprietary style attached to an issue is very unlikely to be actioned, and about a third of the time the minimal reproduction shows the problem was in the harness — a private method that moved, a fixture that encoded an internal detail — which is much better to discover before filing.

Frequently asked questions

How often should a map library be upgraded if every upgrade causes churn?

More often than teams expect, precisely because the churn is proportional to the gap. Two majors of accumulated change produce a diff nobody can triage, so upgrades get deferred further, which makes the next one worse. A cadence of one evaluated upgrade per release cycle keeps each diff small enough that the regressions are visible in it, and the visual suite is what makes that cadence affordable.

Should the old version cell be deleted after the upgrade lands?

Keep it for one release cycle, then let the retention policy move it down a tier. Its value is answering “did this look different before the upgrade” during the period when someone might still ask, which is the weeks immediately afterwards. Beyond that the question is rare enough that an archived copy is fine.

What if the upgrade changes so much that grouping does not help?

That usually means the grouping features are too fine rather than that the change is too large. Widening the coverage buckets and dropping the colour-delta term will collapse a frame-wide rendering change into one row. If it genuinely does not group — different regions changing in different ways with no common signature — the upgrade is better evaluated scenario class by scenario class, capturing only labels first, then only fills, and reviewing each pass separately.

Does this apply to style specification changes as well as library upgrades?

Yes, and the mechanics are identical: a style version behaves like a library version, and the style hash is already in the baseline key. The difference is who reviews it — a style change goes to the cartography owner as a deliberate edit, whereas a library upgrade goes to whoever owns the dependency, with the cartography owner consulted on whichever groups turn out to be rendering changes.

How do library upgrades interact with the cross-browser matrix?

They multiply, which is the reason to keep the engine axis as small as it can honestly be. An upgrade evaluated across three engines produces three two-cell comparisons, and each engine’s churn is genuinely different because each renders the library’s output its own way. The mitigation is the same one described in Cross-Browser Baseline Matrix: collapse the axes that provably agree before the upgrade, not during it.