Adding a Dead-Man's Switch to a Scheduled Map Rebuild

Part of the Pipeline Observability & Alerting guide.

Operative rule: send the ping as the last statement of the successful path — never from a finally block, and never before the publish, or the switch stops meaning “the map was updated” and starts meaning “a process ran”.

Why Absence Needs Its Own Monitor

Every alert that fires on an event shares one blind spot: an event that never happens produces no alert. A scheduler that was disabled for repository inactivity, a worker that exited during a deploy, a cron entry lost when a host was rebuilt — each of these produces total silence, and silence is indistinguishable from a healthy quiet night.

A dead-man’s switch inverts the logic. An external monitor expects a signal at a known cadence; if the signal fails to arrive within the grace period, it alerts. It is the only mechanism in a monitoring stack that treats nothing happening as an event.

How a dead-man's switch detects a stopped pipeline Pings arrive after each successful nightly run. When a run does not happen, no ping arrives. The monitor waits out the grace period, which covers the interval plus the worst run duration, and then alerts — detecting a failure that produced no error anywhere. Three healthy nights, then nothing — and the monitor notices ping Mon 02:41 ping Tue 02:38 ping Wed 02:44 expected Thu — nothing grace period alert No exception was raised, no job failed, no log line appeared — the scheduler simply stopped firing. Error-based alerting cannot see this. Freshness alerting sees it eventually. This sees it tonight.

Production-Ready Implementation

from __future__ import annotations

import os
import urllib.error
import urllib.request

PING_URL = os.environ.get("REBUILD_PING_URL")     # unset in dev — pings are skipped
PING_TIMEOUT = 10


def ping_success(run_id: str) -> None:
    """Signal a COMPLETED, PUBLISHED rebuild. Never called from a finally block.

    A failed ping must not fail the run: the map is already published, and the
    monitor's own outage is not a data incident.
    """
    if not PING_URL:
        return
    request = urllib.request.Request(
        f"{PING_URL}?run_id={run_id}", method="POST", data=b""
    )
    try:
        with urllib.request.urlopen(request, timeout=PING_TIMEOUT) as response:
            if response.status >= 400:
                print(f"warn: ping returned {response.status}")
    except (urllib.error.URLError, TimeoutError) as exc:
        print(f"warn: ping failed: {exc}")          # logged, never raised


def rebuild(record) -> None:
    fetch_source()
    transform()
    if not validation_gates_pass():
        record.outcome = "aborted"
        record.abort_reason = "validation"
        return                                       # NO ping — nothing was published
    record.published_path = publish()
    purge_changed_tiles()
    record.outcome = "ok"
    ping_success(record.run_id)                      # last statement, success only

The two comments in that code are the whole design. An abort must not ping, because the map was not updated and the switch exists to detect exactly that. And a failed ping must not fail the run, because the monitoring system going down is not a reason to mark a successful publish as broken.

Sizing the Grace Period

Too tight and the switch pages someone every time the runner queues for twenty minutes. Too loose and a dead pipeline sits unnoticed through a working day. The arithmetic is simple and worth doing explicitly rather than guessing.

Adding up a grace period For a nightly pipeline the grace period is the twenty-four hour interval plus a worst queue delay of about twenty minutes plus a worst run duration of about forty minutes plus a safety margin of an hour, giving roughly twenty-six hours. Grace period for a nightly rebuild — add the four terms, do not guess scheduled interval 24 h worst observed queue delay + 20 min worst observed run duration + 40 min safety margin + 1 h ≈ 26 h — anything longer and a dead pipeline survives a full working day unnoticed.

Where the Switch Fits Among Other Alerts

Three alerts, three detection windows A failure alert fires within seconds but is blind to a run that never started. A dead-man's switch fires within the grace period and is blind to a run that succeeds against stale upstream data. A freshness alert fires when the data ages past its threshold and covers what the other two miss. Each one is blind to something the others catch failure alert — fires in seconds blind to: a run that never started the cheapest and least complete of the three dead-man's switch — fires within the grace period blind to: stale upstream data the only one that detects a scheduler that stopped freshness alert — fires when the data ages out blind to: nothing — but it is the slowest

The switch is the cheapest of the three instruments to add and the only one that covers silence, which is why it is worth wiring up on the first day of a pipeline rather than after the first incident that it would have caught.

Verification Steps

  • Disable the schedule and confirm the alert arrives within the grace period, not hours later.
  • Force a validation abort and confirm no ping is sent — an aborted run must look like a missed run.
  • Block the ping endpoint and confirm the pipeline still completes successfully.
  • Confirm each pipeline has its own monitor: a shared endpoint stays healthy while one of its pipelines is dead.
  • Check the alert reaches a human out of hours, on the channel they actually watch at 3 a.m.

Common Errors & Fixes

The switch never fires even when the job is stopped

The ping is in a finally block, a shell trap, or a separate scheduled task. Move it to the last statement of the successful path in the job itself.

The switch fires on slow nights

The grace period is smaller than interval plus worst run. Measure the actual distribution from the run records and set the period above the observed maximum, not above the average.

It fires during planned maintenance

Pause the monitor as part of the maintenance procedure, and make un-pausing part of the same checklist — a permanently paused switch is worse than none, because it looks like coverage.

It fires but nobody knows which pipeline

The ping carries no identity. Give each pipeline its own monitor and include the run id in the ping so the alert can link straight to the record.

Choosing Where the Monitor Lives

The one property a dead-man’s switch must have is independence. A monitor that runs on the same host as the pipeline, in the same scheduler, or behind the same credentials will fail in exactly the circumstances that make it necessary — and a monitoring system that goes down silently alongside the thing it watches is worse than none, because it is mistaken for coverage.

That rules out the arrangement teams reach for first: a second scheduled job that checks whether the first one ran. Both live in the same scheduler, so a scheduler that stops firing takes the check with it. The same applies to a check inside the application: a deploy that breaks the application breaks the alarm.

What works is a monitor that lives somewhere with no shared failure mode — a hosted uptime service, a separate account, or at minimum a different scheduler on different infrastructure. The bar is low: the monitor only has to receive a ping and count time, so almost anything qualifies as long as it fails independently.

It is also worth thinking about who receives the alert and what they can do at the moment it arrives. A dead-man’s switch fires precisely when nobody is watching, which usually means out of hours. The alert should therefore be routed to whatever channel actually reaches a person then, and should carry enough context to allow the recipient to decide whether to act now or in the morning: the pipeline name, how long since the last success, the data’s current age, and how stale the dashboard will be by the start of business.

That last number is what turns a page into a decision. A nightly map that missed one run is usually fine until morning; one feeding an operational process is not. Encoding that judgement in the alert text — “data will be 34 hours old at 09:00” — lets the person on call answer the question without opening a laptop, which is the entire purpose of alerting on absence in the first place.

Gotchas & Edge Cases

  • A pipeline that runs on weekdays only needs a monitor that understands the schedule, or it will alert every Saturday morning until someone silences it permanently.
  • Daylight-saving transitions shift a local-time schedule by an hour twice a year; expressing both the schedule and the grace period in UTC removes the class of problem entirely.
  • A monitor configured with the interval but not the run duration fires while a slow run is still working — the grace period must cover the whole cycle, not just the gap between starts.
  • If the ping is sent from a container that is torn down immediately afterwards, ensure the request completes before exit; a fire-and-forget call in a dying process is frequently never sent.
  • Two pipelines that legitimately share a schedule still need separate monitors, or a failure in the quieter one is masked by the busier one’s ping.