Emitting Structured Run Records from a Python Map Pipeline

Part of the Pipeline Observability & Alerting guide.

Operative rule: write the record in a finally block, so a run that crashes still leaves evidence — a missing record must mean “the process never started”, never “the process failed”.

What a Record Is For

Logs answer “what happened during this run”. Records answer “how does this run compare to the others”, and almost every question worth asking about a pipeline is the second kind. Was tonight slower? Has the feature count been drifting? When did the retry rate start climbing? Which build is currently live? Those are table queries, and they need one row per run.

The record is also what makes an incident short. With it, the first minute of triage is reading four numbers. Without it, the first twenty minutes are finding the right log stream and reconstructing a timeline by hand.

Logs versus records for the questions that actually get asked Asking whether a run was slower than usual means parsing timestamps out of logs, or reading one duration field. Asking whether the feature count dropped means grepping, or comparing two numbers. Asking which build is live means inspecting storage, or reading a published path. Asking when a trend started means an afternoon, or sorting a table. The same four questions, two ways to answer them question from logs from records was tonight slower? parse timestamps read one field did the count drop? grep and hope compare two numbers which build is live? inspect storage read published_path when did this start? an afternoon sort a table

Production-Ready Implementation

from __future__ import annotations

import contextlib
import json
import platform
import time
import uuid
from dataclasses import dataclass, asdict, field
from pathlib import Path


@dataclass
class RunRecord:
    run_id: str
    pipeline: str
    trigger: str
    started_at: float
    finished_at: float | None = None
    outcome: str = "running"
    source_timestamp: float | None = None
    feature_count: int | None = None
    previous_feature_count: int | None = None
    published_path: str | None = None
    abort_reason: str | None = None
    error_type: str | None = None
    stage_seconds: dict[str, float] = field(default_factory=dict)
    versions: dict[str, str] = field(default_factory=dict)


@contextlib.contextmanager
def run_record(pipeline: str, trigger: str, out_dir: Path):
    """Always writes a record — success, abort or crash."""
    record = RunRecord(
        run_id=f"{time.strftime('%Y%m%dT%H%M%SZ', time.gmtime())}-{uuid.uuid4().hex[:6]}",
        pipeline=pipeline,
        trigger=trigger,
        started_at=time.time(),
        versions={
            "python": platform.python_version(),
            "geopandas": _safe_version("geopandas"),
            "pipeline": _git_sha(),
        },
    )
    try:
        yield record
        if record.outcome == "running":
            record.outcome = "ok"
    except Exception as exc:                       # noqa: BLE001 — record everything
        record.outcome = "failed"
        record.error_type = type(exc).__name__
        raise
    finally:
        record.finished_at = time.time()
        out_dir.mkdir(parents=True, exist_ok=True)
        (out_dir / f"{record.run_id}.json").write_text(
            json.dumps(asdict(record), indent=2, sort_keys=True), encoding="utf-8"
        )


@contextlib.contextmanager
def stage(record: RunRecord, name: str):
    started = time.time()
    try:
        yield
    finally:
        record.stage_seconds[name] = round(time.time() - started, 2)


def _safe_version(module: str) -> str:
    try:
        import importlib.metadata as md
        return md.version(module)
    except Exception:                              # noqa: BLE001
        return "unknown"


def _git_sha() -> str:
    sha_file = Path(".git/HEAD_SHA")               # written by the build step
    return sha_file.read_text().strip() if sha_file.exists() else "unknown"

The versions block is the field teams add last and wish they had added first. When a pipeline that worked yesterday fails today with no code change, the first useful question is what moved — and a record that names the library versions and the pipeline commit answers it immediately.

What to Put in a Record, and What to Leave Out

Record fields versus log lines The record holds identity, trigger, outcome, timings, counts, the published path and versions — bounded, comparable values. Logs hold per-feature warnings, retry attempts, request traces and debug output, which are unbounded and only useful while investigating one run. If it is unbounded, it belongs in the log, not the record the record — bounded and comparable run id, pipeline, trigger outcome and abort reason per-stage seconds feature count, this run and last published path and source timestamp library and pipeline versions the log — unbounded detail per-feature validation warnings individual retry attempts HTTP traces and payload sizes tool stdout and stderr debug output behind a flag useful for one run, useless as a series

Where the Records Live

Storing records beside the artifacts they describe is simpler than it sounds and better than the alternatives. A path such as runs/<pipeline>/<run_id>.json in the same object storage gives you retention that matches the artifacts, no new infrastructure, and a natural place for a rollback to look up what it is rolling back to.

Where records sit relative to artifacts Under the storage root, a builds directory holds one immutable directory per run id, a runs directory holds one JSON record per run id, a latest pointer names the live build, and a rolling index file lists the last thirty records for review without listing the bucket. One storage layout, four responsibilities builds/<run_id>/ immutable artifacts — tiles, manifest, style document runs/<run_id>.json one record per run — the observability surface latest.json the only mutable object — names the live build and its record runs/index.json the last thirty records, so the weekly review needs no bucket listing

Verification Steps

  • Raise an exception halfway through a run and confirm a record exists with outcome: failed and an error_type.
  • Confirm every stage in the pipeline appears in stage_seconds, including ones that failed.
  • Confirm previous_feature_count is populated by reading the previous record rather than by a separate query.
  • Check the record is written even when publishing failed — the abort reason is the most valuable field in that case.
  • Confirm records survive the retention policy applied to old builds; they are far smaller and worth keeping longer.

Common Errors & Fixes

Records exist only for successful runs

The write is at the end of the happy path. Move it into a finally block, as the context manager above does.

Every record shows the same run id

The id is generated at import time rather than per run. Generate it inside the context manager, and include a random suffix so two runs starting in the same second cannot collide.

The record is huge

Something unbounded is being recorded — usually a list of every warning. Record the count and put the detail in the log.

Timestamps disagree with the CI system’s

The record is using local time. Use UTC everywhere, and stamp the run id with it too so records sort chronologically by filename.

Making Records Readable Without a Dashboard

A directory of JSON files is a database only if something can query it. Most teams do not want to run one for this, and they do not have to: a rolling index file and a short script cover every question that actually gets asked.

The index is simply the last thirty records concatenated into one array, rewritten at the end of each run. It costs one small object write and removes the need to list a bucket or fetch thirty files to answer “how have the last few weeks looked”. Because it is small and immutable-per-write, it can be fetched from anywhere, including from a page.

The script is a dozen lines that prints the index as a table: date, outcome, duration, feature count, delta against the previous run. Run it before a weekly review, or after an incident, and the shape of the problem is usually visible in the first screen. The value is not sophistication — it is that the numbers are already in one place, in the same units, for every run.

If a proper observability stack is available, the records fit naturally into it: each one is a single structured event with a timestamp and a handful of numeric fields, which is exactly the shape metrics and log platforms expect. Emitting them to both places is reasonable, provided the object-storage copy remains authoritative, because that copy survives changes of tooling and retention policy in a way that a vendor’s storage often does not.

The one thing worth avoiding is a bespoke web interface for run history. It is an appealing project and it is almost never worth the maintenance: the audience is small, the questions are few, and a table printed in a terminal answers them as well as a page that now has to be kept working. Spend the effort on the records themselves — more fields, longer retention, better naming — where it compounds.

Gotchas & Edge Cases

  • Records must never contain credentials, tokens or full source URLs with query parameters; they are frequently copied into tickets and chat, where a signed URL becomes a leaked one.
  • A record written before the publish completes describes an intention, not an outcome. Write it once, at the end, from data accumulated during the run.
  • Floating-point durations should be rounded on write; unrounded values make a diff between two records unreadable for no additional precision.
  • Sorting keys on serialisation makes records diffable, which turns “what changed between these two runs” into a one-line command.
  • Retention for records should outlive retention for artifacts. They are tiny, and the question they answer — when did this behaviour start — is usually asked about last quarter.
  • If several pipelines write into one directory, include the pipeline name in the filename as well as in the body; directory listings are how these are found in practice.