Computing GeoJSON Feature Deltas with geopandas

Part of the Incremental Data Processing guide.

Operative rule: reduce every feature to a single fingerprint — a hash of its canonical WKB geometry plus its sorted attributes — keyed on a stable real-world id, then diff the two snapshots on (id, hash) so added, modified, and removed features fall out in one pass.

How the Delta Computation Works

An incremental pipeline only wants to touch what changed. To know what changed between last night’s dataset and today’s, you compare two snapshots — but a naive comparison of geometry objects is both slow and unreliable. Shapely geometries do not compare cleanly on floating-point coordinates, column order varies between exports, and iterating row by row on tens of thousands of features is a bottleneck. The robust approach is to collapse each feature into two values: a stable id that identifies the real-world thing across snapshots, and a content hash that captures everything about the feature’s current state.

The content hash is computed from the geometry’s canonical Well-Known Binary (WKB) representation combined with the feature’s attributes serialised in a deterministic order. Two features with the same id are identical if and only if their hashes match. This turns the whole diff into a set operation. Using geopandas 0.14+ you build a small frame of (id, hash) for each snapshot, then do a single pandas.merge with indicator=True: ids present only on the right are added, ids present only on the left are removed, and ids present on both but with different hashes are modified. The output is a compact change set you can feed into the next stage — appending features to MBTiles or a bounding-box-scoped re-tile — instead of rebuilding the entire map. Getting the geometry hash right also depends on both snapshots sharing one coordinate system; if they do not, normalise first with sound CRS & projection management.

GeoDataFrame delta by id and geometry hash Old and new snapshots are each reduced to id plus content-hash pairs. A merge with indicator classifies ids: right-only are added, left-only are removed, and matched ids with differing hashes are modified. Old snapshot GeoDataFrame A New snapshot GeoDataFrame B (id, hash) pairs WKB + sorted attrs merge indicator outer join added modified removed

Production-Ready Implementation

The routine below loads two GeoJSON snapshots as GeoDataFrames, fingerprints each feature, and returns three frames: added, modified, and removed. It normalises geometry to canonical WKB via shapely, sorts attribute keys so column order never affects the hash, and rounds coordinates to a fixed grid so sub-millimetre float noise does not register as a change.

# geojson_delta.py — classify feature changes between two snapshots.
# Requires: geopandas>=0.14, shapely>=2.0, pandas.

from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from pathlib import Path

import geopandas as gpd
import pandas as pd
from shapely import set_precision  # shapely 2.x
from shapely.geometry.base import BaseGeometry


@dataclass(frozen=True)
class Delta:
    added: gpd.GeoDataFrame
    modified: gpd.GeoDataFrame
    removed: gpd.GeoDataFrame


def _feature_hash(geom: BaseGeometry, attrs: dict, *, grid: float = 1e-7) -> str:
    """
    One fingerprint per feature: canonical WKB of a precision-reduced geometry
    plus attributes serialised with sorted keys. Same geometry + same attrs
    always yields the same digest.
    """
    snapped = set_precision(geom, grid)               # kill sub-grid float noise
    hasher = hashlib.sha256()
    hasher.update(snapped.wkb)                         # canonical binary geometry
    hasher.update(json.dumps(attrs, sort_keys=True, default=str).encode())
    return hasher.hexdigest()


def _fingerprint(gdf: gpd.GeoDataFrame, id_col: str) -> pd.DataFrame:
    """Reduce a GeoDataFrame to a (id, hash) frame keyed on a stable id."""
    attr_cols = [c for c in gdf.columns if c not in (id_col, gdf.geometry.name)]
    hashes = [
        _feature_hash(row.geometry, {c: getattr(row, c) for c in attr_cols})
        for row in gdf.itertuples(index=False)
    ]
    return pd.DataFrame({id_col: gdf[id_col].to_numpy(), "_hash": hashes})


def compute_delta(
    old: gpd.GeoDataFrame, new: gpd.GeoDataFrame, id_col: str
) -> Delta:
    """Return added / modified / removed feature sets between two snapshots."""
    if old.crs != new.crs:
        raise ValueError(f"CRS mismatch: {old.crs} vs {new.crs}; reproject first")

    fp_old = _fingerprint(old, id_col)
    fp_new = _fingerprint(new, id_col)

    merged = fp_old.merge(
        fp_new, on=id_col, how="outer",
        suffixes=("_old", "_new"), indicator=True,
    )

    added_ids = merged.loc[merged["_merge"] == "right_only", id_col]
    removed_ids = merged.loc[merged["_merge"] == "left_only", id_col]
    both = merged[merged["_merge"] == "both"]
    modified_ids = both.loc[both["_hash_old"] != both["_hash_new"], id_col]

    return Delta(
        added=new[new[id_col].isin(added_ids)].copy(),
        modified=new[new[id_col].isin(modified_ids)].copy(),
        removed=old[old[id_col].isin(removed_ids)].copy(),
    )


if __name__ == "__main__":
    old = gpd.read_file(Path("snapshots/incidents_prev.geojson"))
    new = gpd.read_file(Path("snapshots/incidents_curr.geojson"))

    delta = compute_delta(old, new, id_col="incident_id")
    print(
        f"added={len(delta.added)} "
        f"modified={len(delta.modified)} "
        f"removed={len(delta.removed)}"
    )

The identity you diff on determines what a “change” even means. Comparing whole rows is simple and wrong the moment a float is rounded differently; comparing a stable key plus a normalised geometry hash is what makes deltas reproducible across runs.

Three ways to key a delta, and what each one detects Whole-row equality detects everything but reports false changes whenever coordinate precision or field order shifts. A primary key with an attribute hash detects attribute edits reliably but misses geometry moves entirely. A primary key with a rounded, normalised geometry hash detects both and stays stable across serialisation round-trips. What you hash decides which changes you can see whole-row equality detects: everything · false positives: constant a re-export with different float precision reports the entire dataset as changed stable id + attribute hash detects: attribute edits, inserts, deletes · misses: geometry moves a feature that moves keeps its attributes, so nothing is flagged and the old tiles keep a ghost stable id + attribute hash + normalised geometry hash detects: all four delta categories · stable across serialisation round-trips round coordinates to the precision your tiles actually resolve before hashing — 6 decimals is ~10 cm

Alternative Variants

Vectorised hashing for large frames

itertuples is fine up to a few tens of thousands of features. Beyond that, hash the WKB column directly with a vectorised apply, which avoids per-row Python attribute access:

import geopandas as gpd
import pandas as pd

def fast_fingerprint(gdf: gpd.GeoDataFrame, id_col: str, attr_cols: list[str]) -> pd.DataFrame:
    wkb = gpd.GeoSeries(gdf.geometry).to_wkb()                    # bytes per row
    attrs = gdf[attr_cols].astype(str).agg("|".join, axis=1)     # stable string
    combined = pd.Series(wkb).astype(bytes) + attrs.str.encode("utf-8")
    hashes = combined.map(lambda b: __import__("hashlib").sha256(b).hexdigest())
    return pd.DataFrame({id_col: gdf[id_col].to_numpy(), "_hash": hashes.to_numpy()})

What each change class costs downstream

Change class Detection rule Downstream action
Added id in new only Insert feature; tile any covering z/x/y
Modified id in both, hash differs Replace feature; re-tile its footprint
Removed id in old only Delete feature; re-tile its former footprint
Unchanged id in both, hash equal No action — skip entirely

The unchanged bucket is the whole point: in a typical nightly refresh it is the overwhelming majority, and skipping it is what makes the pipeline incremental rather than a full rebuild.

The delta report worth logging

A delta run should end by printing the same five numbers every time. Watching their ratio across runs catches upstream problems long before anyone notices a wrong map — a sudden surge of deletes almost always means a truncated export, not a real change.

The five counters every delta run should print A healthy nightly run shows about ninety-six per cent unchanged, two per cent inserted, one and a half per cent attribute-updated, a fraction of a per cent moved and a fraction deleted. A spike in deletes usually means a truncated upstream export; a spike in inserts usually means the identity key changed. A healthy nightly delta profile — and what a spike in each counter means unchanged 96 % a drop here means the hash inputs changed inserted 2 % a spike means the identity key moved attribute updated 1.5 % the normal shape of an operational feed geometry moved 0.4 % each one dirties two tile sets, not one deleted 0.1 % a surge here means a truncated export — abort the run

Verification Steps

  • Run the delta on two identical snapshots and confirm all three output frames are empty — any non-empty result means a non-deterministic hash input.
  • Modify one attribute of a single feature, rerun, and confirm exactly one feature lands in modified and nothing in added or removed.
  • Move one feature’s geometry by more than the precision grid and confirm it appears in modified.
  • Confirm added, modified, and removed counts reconcile: len(new) == unchanged + added + modified and len(old) == unchanged + removed + modified.
  • Assert both snapshots share a CRS before diffing; the routine raises on mismatch by design.

Common Errors & Fixes

Every feature reports as both added and removed

The id column is not stable across snapshots — you are keying on the row index, an export sequence number, or an auto-increment that resets each build. Switch to a value tied to the real-world feature, such as a database primary key or a domain identifier, so the same feature carries the same id in both snapshots.

Unchanged features show up as modified

Float precision or attribute serialisation differs between exports. Two GeoJSON writers can emit the same point as -74.006 and -74.0060000001. Snap geometries to a fixed precision grid with shapely.set_precision before hashing, and serialise attributes with sort_keys=True and a fixed default so key order and numeric formatting cannot vary.

CRS mismatch error raised on merge

The two snapshots are in different coordinate systems, so their WKB bytes differ even for the same location. Reproject one frame with gdf.to_crs(...) to match the other before computing the delta. Standardising both snapshots to one CRS at ingest avoids the problem entirely.

Modified features detected but their geometry looks identical

An invisible attribute changed — a timestamp column, a computed field, or a source-system audit value that updates on every export even when the feature is unchanged. Exclude volatile bookkeeping columns from the attribute set you hash, keeping only the fields that meaningfully describe the feature.