Pipeline Observability & Alerting

Part of the Data Refresh & Automation Pipelines guide.

The characteristic failure of an automated map is not a crash. It is a dashboard that looks perfectly normal and has been showing last Tuesday’s data for nine days, because an upstream export started returning an empty file and the pipeline dutifully published it. Observability for these systems is therefore not about tracing every step — it is about answering one question continuously: is the map a reader loads right now built from data that is recent enough to act on? Everything in this guide serves that question.

Prerequisites

Step 1 — Emit One Structured Run Record

Free-text logs are for debugging a run you are already watching. What you need for observability is one machine-readable record per run, written at the end, containing everything needed to answer questions later without re-reading logs.

from __future__ import annotations

import json
import time
from dataclasses import dataclass, asdict, field
from pathlib import Path


@dataclass
class RunRecord:
    """One row per rebuild — the unit of observability for the pipeline."""

    run_id: str
    trigger: str                       # "schedule" | "webhook" | "manual"
    started_at: float
    source_timestamp: float | None = None   # when the DATA was produced
    finished_at: float | None = None
    outcome: str = "running"           # "ok" | "aborted" | "failed"
    feature_count: int | None = None
    previous_feature_count: int | None = None
    tiles_written: int = 0
    tiles_purged: int = 0
    published_path: str | None = None
    abort_reason: str | None = None
    stage_seconds: dict[str, float] = field(default_factory=dict)

    def duration(self) -> float | None:
        if self.finished_at is None:
            return None
        return self.finished_at - self.started_at

    def count_delta_pct(self) -> float | None:
        if not self.previous_feature_count or self.feature_count is None:
            return None
        prev = self.previous_feature_count
        return (self.feature_count - prev) / prev * 100.0

    def write(self, directory: Path) -> Path:
        directory.mkdir(parents=True, exist_ok=True)
        path = directory / f"{self.run_id}.json"
        path.write_text(json.dumps(asdict(self), indent=2), encoding="utf-8")
        return path

Two fields in that record do most of the work. source_timestamp is the moment the data was produced upstream, not the moment your job ran — it is what freshness is actually measured from. And previous_feature_count turns every run into a comparison, which is how a truncated export becomes visible without anyone configuring a threshold by hand.

What each field in the run record is for Is the data fresh is answered by the source timestamp rather than the run time. Did anything change is answered by comparing feature count with the previous run. Where did the time go is answered by per-stage seconds. What is live right now is answered by the published path. Why did it stop is answered by the abort reason, which must name the gate that refused. During an incident you ask five questions — the record answers all of them "is the map showing recent data?" source_timestamp — never started_at "did anything actually change?" feature_count vs previous_feature_count "where did the time go?" stage_seconds — fetch, transform, tile, publish "what is live right now?" published_path — the versioned artifact readers receive "why did it stop?" abort_reason — name the gate, not the exception

Step 2 — Publish Freshness as a First-Class Metric

Freshness is the one number that describes the reader’s experience, and it is not the same as build recency. A pipeline that ran four minutes ago from a source that was last updated eleven days ago is producing an eleven-day-old map. Compute the age from source_timestamp, publish it alongside the data so the dashboard itself can display it, and alert on it.

def freshness_seconds(record: RunRecord, now: float | None = None) -> float | None:
    """Age of the DATA in the published artifact, not the age of the run."""
    if record.source_timestamp is None:
        return None
    return (now if now is not None else time.time()) - record.source_timestamp


def freshness_status(age: float | None, warn_after: float, page_after: float) -> str:
    if age is None:
        return "unknown"        # treat as a failure, not as healthy
    if age >= page_after:
        return "critical"
    if age >= warn_after:
        return "warning"
    return "ok"

Publishing that value into the map artifact has a useful side effect: it lets the dashboard show “data as of …” in the interface. Readers stop asking whether the map is current, and when something does go wrong they see it before you do — which is uncomfortable, but far better than the alternative where nobody notices for a week.

Step 3 — Add a Dead-Man’s Switch

Every alert built on events shares a blind spot: no events means no alerts. A scheduler that stops firing, a runner that is disabled for inactivity, a queue consumer that exits quietly — all of them produce perfect silence, and silence looks exactly like success.

The remedy is an external monitor that expects a ping. After a successful run the pipeline calls a monitoring endpoint; if the call does not arrive inside the expected window, the monitor alerts. The critical design detail is that the ping must be last and must be conditional on the publish having actually happened, otherwise it degrades into a liveness check for the runner rather than a correctness check for the map.

Which monitor catches which failure An error alert catches a crashing job but not a job that never started. A success ping with a dead-man's switch catches both, because it alerts on the absence of the ping. A freshness alert catches the case where runs succeed but the data behind them is stale, which neither of the other two can see. Failure mode versus the monitor that sees it failure mode error alert dead-man's switch freshness alert job crashes mid-run caught caught caught, late scheduler stops firing invisible the only one caught, late runs succeed, upstream is frozen invisible invisible the only one You need all three — each is blind to a failure the others catch.

Step 4 — Alert on Symptoms, Log Everything Else

Alert fatigue is a design failure, not a discipline failure. If a channel produces routine noise, humans stop reading it within about two weeks, and the one alert that mattered arrives into a muted room. The discipline that survives contact with reality is to page only on things a reader would notice, and log the rest.

Two alerts qualify for paging in almost every map pipeline: data older than the agreed threshold, and a publish that failed after its retries. A slow run, a retried fetch, a purge that returned a rate-limit response — these are all interesting, none of them are urgent, and all of them belong in the run record where they can be reviewed weekly.

Every alert should also carry enough context to act without opening a laptop: which pipeline, how stale, what the last successful run was, and a direct link to the run record. An alert that says only “rebuild failed” costs the responder ten minutes before they have learned anything.

Step 5 — Verify the Alerting Path Deliberately

An untested alert is a hypothesis. At least once, break each failure mode on purpose in a staging pipeline and confirm the alert arrives: stop the scheduler and wait for the dead-man’s switch; point the source at a frozen file and wait for the freshness alert; revoke the publish credential and confirm the failure alert fires after retries. Then check the arrival channel out of hours, because notification routing that works during the day frequently does not at 3 a.m.

Step 6 — Instrument Each Stage, Not Just the Run

A run-level record tells you that something took nineteen minutes; a stage-level record tells you which nineteen. Timing each stage separately costs almost nothing and turns performance questions into readable answers, because the shape of the breakdown identifies the problem before anyone opens a profiler.

The stages worth timing separately are the ones with different failure characteristics: fetching from upstream, transforming and reprojecting, tiling, publishing, and purging. A fetch that grew from ten seconds to four minutes points at an upstream system under load. A transform that grew points at data volume or at a geometry-repair path that is being hit more often. A tiling step that grew points at density — usually a new area of coverage rather than a general increase. A publish that grew points at the network or at object-storage throttling. Without the split, all four look the same: “the job is slower”.

import contextlib
import time


@contextlib.contextmanager
def stage(record: RunRecord, name: str):
    """Time one stage and record it, whether or not it raises."""
    started = time.time()
    try:
        yield
    finally:
        record.stage_seconds[name] = round(time.time() - started, 2)


def rebuild(record: RunRecord) -> None:
    with stage(record, "fetch"):
        raw = fetch_source()
    with stage(record, "transform"):
        gdf = transform(raw)
        record.feature_count = len(gdf)
    with stage(record, "tile"):
        record.tiles_written = write_tiles(gdf)
    with stage(record, "publish"):
        record.published_path = publish()
    with stage(record, "purge"):
        record.tiles_purged = purge_changed()

Keeping the finally block means a stage that raises still records its duration, which is exactly the case you most want to see: a fetch that failed after ninety seconds tells a very different story from one that failed instantly.

Step 7 — Review Weekly, Not Only During Incidents

Alerts catch cliffs. What they cannot catch is slow rot: a run duration creeping up by four per cent a week, a retry count that has quietly doubled since spring, a feature count drifting downward because an upstream filter changed. None of these will ever cross a threshold on the night they matter, and all of them are obvious in a table of the last thirty runs.

A ten-minute weekly review of the run records is the cheapest reliability practice available for a pipeline of this kind. Read the durations, the counts and the retry totals in sequence and ask one question: is anything trending? Anything that is gets a ticket while it is still cheap to fix, rather than a page when it finally crosses a line at three in the morning. Teams that do this find that the number of genuine incidents falls, and the ones that remain are caused by external events rather than by their own accumulated drift.

Step 8 — Make the Dashboard Report Its Own Health

The last piece of instrumentation belongs in the product rather than in the operations stack. A small, permanent line in the interface — “data as of 05 August 2026, 02:41 UTC” — turns every reader into a monitor, and costs one string in the build manifest.

Two refinements make it genuinely useful rather than decorative. Render the age relative to now on the client, so a reader who leaves the tab open overnight sees the value grow rather than a fixed timestamp that quietly becomes a lie. And give the line a visible state change once the age passes the agreed warning threshold, so staleness is legible at a glance instead of requiring arithmetic. A reader who can see that the map is nine days old will not act on it, which is precisely the outcome the whole observability effort exists to produce.

This also changes the conversation when something does break. Instead of a report that says “the map looks wrong”, you get one that says “the map says it is eleven days old” — which names the failure, points at the pipeline rather than the renderer, and can be triaged in seconds.

What belongs on a page and what belongs in a channel

Instrumentation output splits cleanly into three audiences, and mixing them is what produces both alert fatigue and dashboards nobody reads. Deciding the destination per signal, once, keeps each channel worth attending to.

Three destinations for three audiences The dashboard itself carries the data's age, for readers. A weekly review surface carries durations, counts and retry rates, for the pipeline's maintainers. The alert channel carries only stale data and failed publishes, for whoever is on call. Three audiences — route each signal to exactly one of them the dashboard → readers the data's age, in words, always visible turns every reader into a monitor, at the cost of one string the review surface → maintainers durations, counts, retries, trends read weekly, not during incidents — this is where slow rot is caught the alert channel → whoever is on call two things only: stale data, failed publish anything else here trains the channel away within a fortnight

Verification & Smoke-Test

  • Record completeness — assert every run writes a record with a non-null outcome, even when it aborts. A missing record is itself a signal.
  • Freshness arithmetic — set source_timestamp to a known past value and confirm the published age matches it to the second.
  • Switch timing — disable the schedule and confirm the dead-man’s switch fires within the configured grace period, not hours later.
  • Alert content — confirm the delivered message names the pipeline, the age, the last good run and a link.
  • Weekly review — read the last seven run records end to end; anything you cannot explain is a gap in the instrumentation.

Troubleshooting

Alerts fire every night and everyone ignores them

The threshold is tighter than the pipeline’s real variability. Measure the actual distribution of run durations and freshness over a month and set the threshold above the 99th percentile, not above the average.

The dead-man’s switch never fires even when the job is stopped

The ping is being sent from a wrapper that runs regardless of outcome — a shell trap, a finally block, or a separate cron entry. Move the ping to the last line of the successful path only.

Freshness looks fine but the map is stale

The pipeline is stamping source_timestamp with its own clock rather than reading it from the upstream payload. If the source provides no timestamp, use the last-modified header of the file it fetched; if it provides nothing at all, treat that as a data-quality defect and raise it upstream.

A failed run leaves the pipeline permanently in “running”

The record is written only on success. Write it in a finally block with outcome="failed" and the exception type — the record must always exist, and its absence should be treated as a failure by the reviewer.

Two pipelines share one monitor and neither failure is visible

A single ping endpoint receiving traffic from several pipelines stays healthy as long as any one of them is running. Give every pipeline its own monitor and its own expected interval, even when they run on the same schedule, or the busiest one masks the rest.

Gotchas & Edge Cases

  • Feature-count comparisons need a floor: a dataset that legitimately drops to zero features overnight will trip a percentage-based check every time.
  • Timestamps must be timezone-aware end to end; a naive local timestamp compared against a UTC one produces freshness values that are wrong by exactly the offset.
  • A retry that eventually succeeds should not page, but it should be counted — a slowly rising retry rate is the earliest warning an upstream system is degrading.
  • Monitoring endpoints are a dependency too: if the ping fails, the run should still be considered successful, and the failure logged.
  • Keep run records longer than you think you need. Answering “when did this start” is usually a question about last quarter, not last night.