Alerting on Feature-Count Anomalies After a Rebuild

Part of the Pipeline Observability & Alerting guide.

Operative rule: run the count check before the publish, and treat a failure as an abort — the previous artifact staying live is the correct outcome, not a fallback.

Why the Count Is the Best Single Signal

Of all the numbers a rebuild produces, the feature count catches the widest range of upstream failures for the least implementation effort. A truncated download halves it. A failed join zeroes a whole layer. A duplicated join doubles it. A schema change that silently drops rows shows up as a step change. None of those raise an exception anywhere — the pipeline works perfectly on the wrong data.

It is not a complete check. A file that arrives with the right number of rows and stale values passes it, which is why freshness is monitored separately. But as one assertion in a build gate, it earns its place several times over.

What a feature-count check catches A truncated download roughly halves the count. A failed join drops a layer's rows to zero. A duplicated join multiplies the count. A schema change that silently drops rows produces a step change. None of these raise an error in the pipeline. Four silent failures, one number that sees them all truncated download count halves — connection dropped mid-transfer, no exception raised failed join a layer drops to zero — key column renamed upstream duplicated join count multiplies — a one-to-many relationship appeared in the source schema change a step change that persists — the most easily missed of the four

Production-Ready Implementation

from __future__ import annotations

import json
import statistics
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class CountPolicy:
    """Bands for a nightly dataset. Tune from the observed series, not by taste."""

    max_drop_pct: float = 15.0      # more than this below the median → abort
    max_rise_pct: float = 25.0      # more than this above → abort (duplicate joins)
    absolute_floor: int = 100       # below this, something is structurally wrong
    baseline_runs: int = 14         # a fortnight of history


def rolling_baseline(records_dir: Path, policy: CountPolicy) -> float | None:
    """Median of the last N successful counts — robust to a single bad night."""
    counts: list[int] = []
    for path in sorted(records_dir.glob("*.json"), reverse=True):
        record = json.loads(path.read_text(encoding="utf-8"))
        if record.get("outcome") != "ok" or record.get("feature_count") is None:
            continue
        counts.append(int(record["feature_count"]))
        if len(counts) >= policy.baseline_runs:
            break
    return statistics.median(counts) if counts else None


def check_count(count: int, baseline: float | None,
                policy: CountPolicy) -> tuple[bool, str]:
    """Return (ok, reason). Called BEFORE publish; a False result aborts the run."""
    if count < policy.absolute_floor:
        return False, f"count {count} below absolute floor {policy.absolute_floor}"
    if baseline is None:
        return True, "no baseline yet — first runs always pass"
    delta_pct = (count - baseline) / baseline * 100.0
    if delta_pct < -policy.max_drop_pct:
        return False, (f"count {count} is {abs(delta_pct):.1f}% below the "
                       f"{policy.baseline_runs}-run median {baseline:.0f}")
    if delta_pct > policy.max_rise_pct:
        return False, (f"count {count} is {delta_pct:.1f}% above the median "
                       f"{baseline:.0f} — check for a duplicated join")
    return True, f"count {count} within band ({delta_pct:+.1f}%)"

The absolute floor matters as much as the percentage bands. A dataset that legitimately shrinks over months will drag its own median down with it, and a purely relative check would eventually accept a count of three. The floor is the backstop that says “whatever the trend, this is not a dataset any more”.

Reading a Real Series

A nightly count series with one truncated night Thirteen nights sit between about twenty-four and twenty-six thousand features, varying by a few per cent. One night falls to about twelve thousand, far outside the fifteen per cent band around the median, which is the signature of a truncated export rather than of real change. Fourteen nights of counts — normal variation, then a truncation ±15 % band around the median 12 100 median ≈ 25 200 features · nightly variation ±3 % The one low night is 52 % below the median — an abort, not an alert to read in the morning. Because the baseline is a median, that night does not drag the band down for tomorrow.

Where the Check Belongs in the Run

The count check as a publication gate Fetch and transform run first. The count check then decides: within the band, the run publishes and swaps the pointer; outside it, the run aborts, records the reason and leaves the previous artifact live for readers. A gate, not a report — it decides whether the publish happens fetch + transform count check against the rolling median within band → publish, swap the pointer, ping outside → abort, record the reason, no ping On abort the previous artifact stays live — readers see yesterday's map, which is correct. Suppressing the ping means the dead-man's switch also fires if the abort repeats, so a persistent upstream problem escalates on its own rather than needing a second alert rule.

Verification Steps

  • Truncate a source file to half its rows in staging and confirm the run aborts before publishing.
  • Duplicate every row and confirm the upper band catches it.
  • Delete the run-record history and confirm the first run passes rather than crashing on a missing baseline.
  • Confirm the abort reason in the record names the numbers, not just “validation failed”.
  • Confirm the previously published artifact is still being served after an abort.

Common Errors & Fixes

The check passes on a day when half the layers are empty

The count is being taken across all layers combined. Check per layer as well as in total; a zeroed layer is invisible in an aggregate that other layers dominate.

The band is too tight and aborts on legitimate growth

The dataset has seasonality the band does not model. Widen it, or compare against the same weekday from previous weeks rather than against a flat median.

One bad night poisons the baseline

The baseline is including aborted or failed runs. Filter to outcome == "ok", as the code above does.

The alert says a count changed but not which dataset

The record does not carry the layer breakdown. Store counts per layer in the run record — it costs a dictionary and saves the first ten minutes of every investigation.

Beyond the Count: Cheap Checks Worth Adding Beside It

The feature count is the best single number, but it is blind to a whole class of problems in which the right quantity of wrong data arrives. Three additional checks cost almost nothing to compute in the same pass and cover most of that gap.

Total geometry length or area is the first. A road network whose feature count is unchanged but whose total length has halved has been truncated in a way the count cannot see — for example a source that now returns simplified geometry, or a clip that removed sections rather than whole features. Comparing the total against the rolling median catches it immediately.

Attribute completeness is the second. Recording the proportion of null values per field turns a silently dropped join into a visible number: the count is right, the geometry is right, and one column is suddenly ninety per cent empty. Because it is a ratio rather than an absolute, it needs no per-dataset tuning to be useful.

Bounding box is the third and cheapest. A dataset’s extent should move slowly or not at all; a sudden jump usually means a CRS regression or a corrupted coordinate, and both are far easier to diagnose from a bounds comparison than from the resulting map. This check also has an unusually low false-positive rate, which makes it a good candidate for blocking rather than merely warning.

Together with the count, those four numbers form a compact data-quality profile that fits in the run record and can be reviewed in a table. Their value comes from being compared over time rather than from any single threshold, which is why they belong in the record even on days when nothing fails.

One caution: resist the urge to add a dozen more. Each check has a false-positive rate, and a gate that blocks publication once a fortnight for reasons nobody can quickly explain gets disabled within a month. Four robust numbers that are trusted are worth far more than twelve that are routinely overridden.

Gotchas & Edge Cases

  • A dataset with strong weekly seasonality will trip a flat band every Monday; compare against the same weekday in previous weeks rather than widening the band until it catches nothing.
  • The first runs after a coverage change legitimately break the baseline. Reset it deliberately, and record why, rather than letting a series of overrides quietly retrain it.
  • Counting features after simplification measures a different thing from counting them before; pick one point in the pipeline and stay there, or the series is not comparable with itself.
  • Multi-layer datasets need per-layer counts as well as a total, because one layer dropping to zero is invisible inside an aggregate the other layers dominate.
  • An abort should still write a run record with the numbers that caused it — the first question during triage is always “how far outside was it?”