Testing a MapLibre GL Major Upgrade Visually

A MapLibre GL major bump moves pixels everywhere, because the library is doing the drawing: symbol placement, text halos, line joins, tile fade timing and anti-aliasing can all change in one release. Compared against a stored baseline the result is uninterpretable, since it mixes the library change with everything that has drifted since that baseline was blessed. Captured as two cells in one run, it becomes a measurement — and the handful of groups that the release notes do not explain are exactly what the exercise exists to find.

This is a task within Map Library Version Upgrade Testing, under Web Map Visual Testing Fundamentals & Toolchains. The grouping it depends on is described in Triaging a Large Map Baseline Re-Bless.

Prerequisites

Step-by-step procedure

1. Capture both versions in one matrix run 

strategy:
  fail-fast: false
  matrix:
    include:
      - { version: '4.7.1', major: '4' }
      - { version: '5.0.0', major: '5' }
steps:
  - uses: actions/checkout@v4
  - run: npm ci
  - run: npm install maplibre-gl@$ --no-save
  - run: npm run capture -- --version-key=maplibre-$
  - uses: actions/upload-artifact@v4
    with:
      name: captures-$
      path: out/captures
The matrix job that captures both library versions in one run A single pipeline run fans out across a version axis with two cells. Both cells install the same application at the same commit, restore the same tile fixtures, and run on the same runner image; the only difference is the map library version installed at the start. Each cell writes its captures under a key that carries its version, so neither overwrites the other. A final job diffs the two cells against each other. A caption states what this buys: the measured difference contains the library change and nothing else, because every other input was identical within the run. One run, two cells, one variable same commit · same fixtures · same runner image · same camera set cell A · maplibre 4.7.1 writes …/maplibre-4/… 120 captures cell B · maplibre 5.0.0 writes …/maplibre-5/… 120 captures diff cell A against cell B

2. Diff the cells against each other 

npx map-diff \
  --baseline out/captures-4.7.1 \
  --candidate out/captures-5.0.0 \
  --report out/upgrade-report.json

Comparing the new version against the stored baseline instead is the mistake that makes upgrades take weeks: the result then contains the library change plus every environment and style drift since that baseline was blessed, and none of it can be separated.

3. Group the report and read it against the release notes

Reading the release notes against the group descriptions Each diff group from the two-cell comparison is placed beside the release-note entry that explains it. A frame-wide symbol placement shift maps to a noted improvement in label collision. A halo rendering change maps to a noted text rendering rewrite. A line-join change on high-zoom roads maps to nothing in the notes and is marked incidental — accepted, but recorded, because nothing else would have found it. A localised failure at high latitude maps to nothing and is marked a regression candidate. A caption notes that the incidental row is the most valuable output of the exercise, because it is a real behaviour change that the release did not mention. Every group gets an entry, or it gets investigated symbol placement shift, frame-wide noted: collision improved halo rendering change on labels noted: text rendering rewrite line joins on z17+ roads not noted — incidental, record it localised failure at 71°N not noted — regression candidate

Read every version’s notes between current and target, not just the target’s. A change introduced in an intermediate release is still new to you, and it is the most common source of a group that appears unexplained.

4. Check the four MapLibre areas that move most between majors

Rather than working purely from the diff, look specifically at the places this library historically changes, because a small change in one of them can hide inside a large group.

  • Symbol placement and collision — a label that used to render and now does not is a content change, not an aesthetic one, and it is easy to accept as part of a frame-wide placement group.
  • Text halo and glyph rasterisation — halos change width or blend differently, which produces a frame-wide wash that masks smaller label changes underneath it.
  • Line joins, caps and casings — visible mainly at high zoom, and often not mentioned in release notes.
  • Raster fade and tile transition timing — if a fade duration default changes, captures taken through a settled gate are unaffected while captures taken on a timer are not, which is a useful signal that a scenario’s gate is weaker than it should be.

5. Narrow every regression candidate before filing it

Reducing a suite failure to a reproduction worth filing A funnel narrows a failing capture into a reportable case. The full application capture is the widest step and is not filable, because it carries a proprietary style and an entire application. Removing the application leaves the style and the camera on a bare map page. Trimming the style to the smallest set of layers that still shows the difference removes most of what a maintainer would have to read. Replacing the tiles with a small synthetic fixture removes the data dependency. The narrowest step is a single HTML file with a handful of layers and inline data, which is what an upstream issue can act on. A note observes that about a third of the time this process shows the fault was in the harness rather than in the library. Narrow until it is one file, then file it full application capture — not filable bare map page, real style and camera style trimmed to the layers that still show it one HTML file, inline data about a third of the time, narrowing shows the fault was in the harness

A capture from your application with a proprietary style attached to an upstream issue is very unlikely to be actioned. The narrowing is also where harness bugs surface: a fixture that encoded an internal detail, a readiness predicate that called a method which has moved, a style property that was always invalid and used to be ignored.

6. Bless the new cell as one attributed batch

The annotation names the version pair, the release, the reviewer and each accepted group signature. This is what makes the churn attributable a year later, when someone asks why nine hundred baselines all changed on the same day.

7. Switch the default and keep the old cell for a cycle

The suite now compares against the new cell. The old one stays readable for one release cycle, which costs a little storage and removes the need for a restore if the upgrade is reverted.

Verification

Confirm the procedure worked before wiring it into a blocking gate:

Troubleshooting

Symptom Likely cause Fix
The two cells differ in scenarios that render no map at all The new version changed an initialisation API, so the map never constructed and the capture shows an empty container Check the console output from the failing cell; an upgrade that breaks construction is a code change, not a rendering one
Every capture in the new cell times out An event the readiness predicate waits on is no longer emitted, or is emitted in a different order Treat a suite-wide timeout as a signal about event semantics rather than infrastructure, and read the notes for event changes
The diff is enormous and groups into hundreds of rows The signature buckets are too narrow for a frame-wide rendering change Widen the coverage bucket and drop the colour-delta term, per Triaging a Large Map Baseline Re-Bless

Frequently asked questions

Should the two cells run on the same runner or on parallel runners?

Parallel runners are fine provided the image is pinned by digest, which it should be anyway. What matters is that the image, the fixtures and the commit are identical; the machine is not an input once the GL backend is software-rendered. If the suite is not yet pinned to a digest, run both cells sequentially on one runner until it is, because otherwise the comparison contains the runner difference too.

How many majors can safely be skipped?

One at a time is much cheaper than several at once, because the diff from two majors is the union of both changes and the release notes have to be read together. If several have accumulated, upgrading through each in turn — accepting the churn at each step — is usually faster in total than triaging the combined diff, even though it means more re-blesses.

What if the application uses a fork or a pinned patch?

Then the fork’s own diff against upstream is a third axis, and the honest procedure is to evaluate upstream-to-upstream first, then apply the fork and evaluate again. Combining them produces a diff in which a fork conflict and a library regression are indistinguishable, and the fork is the thing least likely to have release notes.

Does the two-cell comparison replace the normal baseline gate?

No — it is an additional measurement taken on the upgrade branch. The normal gate still runs against the current default cell throughout, which is what keeps the branch honest about whether it broke anything unrelated to the upgrade. The two-cell diff answers a different question, and answering it does not remove the need for the first.