Benchmarking Map Frame Rate in a Headless Browser

Part of the Choosing a Renderer: Folium vs MapLibre GL vs PyDeck guide.

Operative rule: measure the tail, not the mean — a map that averages 55 frames per second while dropping to 8 during every pan is a map readers describe as broken.

What to Measure

Perceived smoothness is governed by the worst frames, not the typical ones. A run that renders nine hundred frames at 60 fps and forty at 8 fps has an excellent average and a terrible feel. Three numbers describe the experience honestly: the 95th-percentile frame interval, the count of frames longer than 50 milliseconds, and the longest single frame.

Two further measurements matter for a dashboard rather than a demo: time to first interactive frame, which is what a reader waits through before anything responds, and heap growth across a fixed interaction sequence, which is how layer-switching leaks announce themselves.

Five metrics and what each one reveals The 95th percentile frame interval describes how the map feels during interaction. The count of frames over 50 milliseconds counts visible stutters. The longest frame identifies the worst single hitch. Time to first interactive frame measures the wait before anything responds. Heap growth across a fixed sequence reveals leaks. Record these five; ignore the average p95 frame interval how the map feels while the reader is moving it frames > 50 ms a direct count of visible stutters in the run longest frame usually a style parse, a tile decode or a buffer upload time to first interactive frame what a reader waits through before anything responds heap growth across a fixed sequence — the leak detector, and the reason to script the sequence

Production-Ready Implementation

The script drives a fixed sequence so two runs are comparable, and records frame intervals from inside the page.

// bench.mjs — node bench.mjs <url> <out.json>
import puppeteer from "puppeteer-core";
import fs from "node:fs";

const [url, out] = process.argv.slice(2);
const browser = await puppeteer.launch({
  executablePath: process.env.CHROME_PATH,
  headless: true,
  args: ["--no-sandbox", "--enable-gpu", "--use-gl=swiftshader"],
});
const page = await browser.newPage();
await page.setViewport({ width: 1280, height: 900, deviceScaleFactor: 1 });

await page.evaluateOnNewDocument(() => {
  window.__frames = [];
  let last = performance.now();
  const tick = (now) => {
    window.__frames.push(now - last);
    last = now;
    requestAnimationFrame(tick);
  };
  requestAnimationFrame(tick);
});

await page.goto(url, { waitUntil: "networkidle0", timeout: 60000 });
await page.waitForFunction(() => window.__mapReady === true, { timeout: 60000 });

// A fixed, deterministic interaction sequence — no randomness, no timing drift.
const SEQUENCE = [
  { action: "zoomTo", value: 8 }, { action: "panBy", value: [400, 0] },
  { action: "panBy", value: [0, 300] }, { action: "zoomTo", value: 12 },
  { action: "panBy", value: [-400, 0] }, { action: "zoomTo", value: 6 },
];
await page.evaluate(async (steps) => {
  window.__frames.length = 0;                       // discard load frames
  for (const step of steps) {
    if (step.action === "zoomTo") window.map.zoomTo(step.value, { duration: 900 });
    if (step.action === "panBy") window.map.panBy(step.value, { duration: 900 });
    await new Promise((r) => setTimeout(r, 1100));  // let each move complete
  }
}, SEQUENCE);

const frames = await page.evaluate(() => window.__frames.slice());
const heap = await page.evaluate(() => performance.memory?.usedJSHeapSize ?? null);
const sorted = [...frames].sort((a, b) => a - b);
const p = (q) => sorted[Math.floor(sorted.length * q)] ?? 0;

fs.writeFileSync(out, JSON.stringify({
  frames: frames.length,
  p50_ms: Number(p(0.5).toFixed(2)),
  p95_ms: Number(p(0.95).toFixed(2)),
  worst_ms: Number((sorted.at(-1) ?? 0).toFixed(2)),
  long_frames: frames.filter((f) => f > 50).length,
  heap_bytes: heap,
}, null, 2));

await browser.close();

Setting window.__mapReady from the page when the renderer reports its first idle is what makes the measurement start at a comparable moment. Without it the script begins timing during tile loading on one run and after it on another, and the numbers move for reasons that have nothing to do with the change under test.

Making the Result Trustworthy

Controlling the four sources of benchmark noise Different machines make absolute numbers meaningless, so compare only runs from the same runner. Background load skews results, so run benchmarks alone. A non-deterministic interaction sequence changes what is measured, so script it exactly. Network variance changes tile timing, so serve fixtures locally. Compare like with like, or do not compare at all machine variance compare runs from one runner class only — never a laptop against CI background load run the benchmark alone, never in parallel with other jobs interaction variance script the exact sequence — same zooms, same pans, same durations network variance serve fixtures from localhost so tile latency is constant

Using It as a Build Gate

A benchmark that nobody reads is a slow test. The value comes from comparing against the previous build and failing when the tail gets worse by more than the run-to-run noise — which you can measure by running the same build twice.

Turning the benchmark into a gate The new run's ninety-fifth percentile and long-frame count are compared against the stored baseline. Within the noise margin the build passes and the baseline is updated. Beyond it the build fails with both numbers in the message, so the regression is visible without opening an artifact. Measure noise first, then set the margin above it 1 · run the same build twice — the spread between them is your noise floor typically 10–20 % on shared CI hardware 2 · fail when p95 or long-frame count exceeds the baseline by more than that report both numbers in the failure message, not a link to an artifact 3 · update the baseline only on a passing run of the default branch otherwise a slow build quietly becomes the new normal

Verification Steps

  • Run the benchmark twice on an unchanged build and confirm the spread is within your assumed noise margin.
  • Add a deliberately expensive layer and confirm the gate fails.
  • Confirm frames recorded during load are discarded, so the numbers describe interaction only.
  • Confirm heap growth over the fixed sequence is near zero on a build with no known leaks.
  • Confirm the run fails loudly if __mapReady never becomes true, rather than reporting a suspiciously good result.

Common Errors & Fixes

Frame intervals are all around 16 ms and nothing ever regresses

Nothing is actually rendering — the sequence ran before the map was ready, or the animations completed instantly because durations were not set.

Results swing wildly between runs

The runner is shared or the benchmark is running alongside other jobs. Isolate it, and raise the margin only after confirming isolation did not fix it.

The headless run is far slower than a real browser

Software rendering is in use. That is acceptable for relative comparison as long as every run uses it, but do not publish the numbers as absolute performance.

The gate blocks a change that is genuinely a trade

Record the reason in the baseline update rather than removing the gate. A deliberate slowdown accepted for a feature is fine; an accidental one is what the gate exists to catch.

Measuring the Load Path as Well as the Interaction

Frame rate describes the map once it is running. It says nothing about the several seconds before that, which for most readers is the more memorable part of the experience — particularly on a dashboard they open once a day rather than keep open all afternoon.

Three load-path numbers are worth capturing in the same run. Time to the first rendered frame tells you when the reader stops looking at an empty container. Time to first interactive frame — when panning actually responds — is usually noticeably later, because style parsing and initial tile decoding are still occupying the main thread. And total bytes transferred before that point is the number that connects the experience back to something you can change, since it is dominated by decisions about payload rather than by rendering.

Capturing them costs very little once the harness exists. The page already signals readiness for the interaction phase; adding two earlier marks and reading the browser’s own performance entries gives the whole picture from one run. Recording all of it into the same JSON artifact means a regression in load time is caught by the same gate that watches frame time, instead of being noticed months later by a reader.

It is also worth separating cold and warm measurements. A first visit pays for the library, the style, the fonts and the first tiles; a return visit with a warm cache pays for almost none of it. Both matter, and they respond to different fixes — payload work improves the first, caching work improves the second. Running the harness twice, once with a fresh profile and once against a warmed one, distinguishes them cleanly.

The last piece of discipline is to keep the benchmark honest about what it does not measure. A headless run with software rendering tells you nothing reliable about how the map performs on a mid-range phone, which is where a large share of readers actually are. Treat the harness as a regression detector, not as a statement of user experience, and complement it with occasional measurements on real hardware.

Gotchas & Edge Cases

  • performance.memory is Chromium-only and reports the whole heap, not the map’s share; treat its trend across a fixed sequence as the signal, never its absolute value.
  • Animation durations must be explicit: a zoom with no duration completes immediately and the run measures almost nothing.
  • Frames recorded while tiles are still loading describe the load path, not interaction; clear the buffer after the ready signal.
  • A benchmark that renders at a device pixel ratio of one measures fewer pixels than a retina reader sees — pin the ratio and state which one the numbers describe.
  • Baselines should be stored per runner class, because a change of CI machine shifts every number and looks exactly like a regression.