Choosing a Scheduler: Celery vs APScheduler vs cron

Part of the Data Refresh & Automation Pipelines guide.

How you run a map-rebuild job matters as much as the rebuild logic itself. The same GeoJSON-to-tiles pipeline can be a five-line crontab entry or a fleet of distributed workers, and choosing wrong means either paying for infrastructure you do not need or discovering at 3 a.m. that a silent failure left stale tiles on the CDN for a week. Three tools cover the entire spectrum. cron is the dead-simple single-box scheduler with no retries and no observability, perfect for a nightly rebuild. APScheduler is an in-process Python scheduler that lives inside one Flask or FastAPI app with persistent jobstores. Celery with a Redis or RabbitMQ broker is the distributed option with retries, concurrency, rate limits, and Flower monitoring for heavy or parallel rebuilds. This guide gives backend and data engineers a framework to size the workload and pick deliberately. The stack components are cron, APScheduler 3.10+, Celery 5.3+, and a redis or rabbitmq broker.

Scheduler Escalation Ladder Diagram showing three escalating tiers of scheduling. cron drives a single box for nightly rebuilds. APScheduler runs inside one Flask or FastAPI process with a jobstore. Celery beat enqueues to a broker that fans out to multiple workers with retries. cron single box · nightly no retries · no metrics APScheduler in-process · Flask/FastAPI jobstore persistence Celery + broker distributed · retries concurrency · Flower Broker Redis / RabbitMQ worker 1 — rebuild region A worker 2 — rebuild region B worker 3 — rebuild region C

Prerequisites

Line up the tooling and the workload facts before choosing a tier, because the right scheduler depends entirely on numbers you have not measured yet:

If rebuilds are triggered by data changes rather than the clock, the event-driven path in Webhook-Triggered Updates may replace or complement the scheduler entirely.

Step 1 — Characterize the Rebuild Job

The scheduler choice collapses to three measurable properties of the job. Measure them first; picking Celery for a 15-second nightly job is as wrong as cron-ing a 200-region parallel rebuild.

  1. Duration. How long does one rebuild take — seconds, minutes, or tens of minutes? Long jobs overlap with the next scheduled run and demand overrun protection.
  2. Frequency. Nightly, hourly, or every few minutes? High frequency plus long duration is the combination that forces concurrency.
  3. Parallelism. Is it one monolithic rebuild, or can it fan out into independent units — per region, per layer, per tile pyramid — that run concurrently? Fan-out is the single strongest signal for Celery.
from dataclasses import dataclass


@dataclass(frozen=True)
class JobProfile:
    """The three properties that decide the scheduler tier."""
    duration_seconds: float
    runs_per_day: int
    parallel_units: int   # 1 = monolithic; >1 = fan-out candidate


def recommend_scheduler(p: JobProfile) -> str:
    """Opinionated mapping from workload shape to scheduler tier."""
    if p.parallel_units > 1 or p.runs_per_day > 24 or p.duration_seconds > 600:
        return "celery"       # distribution, high frequency, or long jobs
    if p.runs_per_day > 1 or p.duration_seconds > 60:
        return "apscheduler"  # app-embedded, needs persistence/overrun guards
    return "cron"             # infrequent, short, self-contained

The decision matrix the function encodes is the reference to keep on the wall:

Dimension cron APScheduler Celery + broker
Scale ceiling single box, few jobs one app process many workers, many hosts
Retries none (manual) basic (misfire_grace_time) first-class (max_retries, backoff)
Observability log files only app logs / event listeners Flower dashboard, task events
Ops overhead near zero low (in-process) high (broker + workers)
Concurrency none thread/process pool distributed, rate-limited

Feed the same job characterisation into your Incremental Data Processing design — a rebuild that only touches changed features is shorter and cheaper, which can pull the recommendation a whole tier down.

Step 2 — Start with cron

For a single box running an infrequent, self-contained rebuild, cron is the correct answer, not a compromise. It ships with the OS, has no moving parts, and a nightly GeoJSON-to-tiles rebuild is exactly what it was built for. The catch is that cron gives you nothing beyond firing the command: no retries, no metrics, no overlap protection. You accept that in exchange for zero operational surface.

# crontab -e  — rebuild tiles nightly at 02:30
# Use ABSOLUTE paths; cron's PATH is minimal and will not find your venv.
30 2 * * * /srv/maps/.venv/bin/python /srv/maps/rebuild.py >> /var/log/maps/rebuild.log 2>&1

# A flock guard prevents a slow run from overlapping the next trigger:
0 * * * * /usr/bin/flock -n /tmp/rebuild.lock /srv/maps/.venv/bin/python /srv/maps/rebuild.py >> /var/log/maps/rebuild.log 2>&1

The flock -n wrapper is the one addition that turns naive cron into something safe for jobs that occasionally run long: it refuses to start a second copy while the first holds the lock. Redirecting both stdout and stderr to a log file is non-negotiable — cron mails output by default, and on a headless box that mail goes nowhere. When you find yourself bolting on retry loops, lock files, and log parsing, that is the signal cron has been outgrown. The single-server trade-offs are weighed in detail in Choosing cron vs APScheduler for Single-Server Dashboards.

Step 3 — Move to APScheduler for App-Embedded Scheduling

When scheduling needs to live inside a long-running Python application — a Flask or FastAPI dashboard that already owns the rebuild code — APScheduler is the natural fit. It runs in-process, so there is no separate service to deploy, and it schedules functions directly rather than shelling out. The critical upgrade over cron is a persistent jobstore: back it with SQLAlchemyJobStore and scheduled jobs survive process restarts and redeploys instead of evaporating.

from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
from apscheduler.triggers.cron import CronTrigger

from rebuild import run_rebuild  # your existing entry point

scheduler = BackgroundScheduler(
    jobstores={"default": SQLAlchemyJobStore(url="sqlite:///jobs.sqlite")},
    job_defaults={
        "coalesce": True,          # collapse missed runs into one
        "max_instances": 1,        # never overlap the same job
        "misfire_grace_time": 300, # still run if fired up to 5 min late
    },
)

scheduler.add_job(
    run_rebuild,
    trigger=CronTrigger(hour=2, minute=30),
    id="nightly_rebuild",
    replace_existing=True,   # idempotent registration across restarts
)
scheduler.start()

The coalesce, max_instances, and misfire_grace_time settings give you the overrun and missed-run handling cron lacks, without a broker. APScheduler suits one app process; its weakness is exactly there — under a multi-worker server like Gunicorn, each worker starts its own scheduler and jobs fire N times. The Flask-specific setup, including how to pin scheduling to a single worker, is walked through in Scheduling Map Rebuilds with APScheduler in Flask.

Step 4 — Adopt Celery for Distributed Retries and Concurrency

When rebuilds must fan out across workers, retry automatically, or run more often than a single process can keep up with, move to Celery with a redis or rabbitmq broker. Celery decouples scheduling from execution: a beat process enqueues tasks onto the broker, and any number of worker processes on any number of hosts pull and run them. That is what makes per-region or per-layer rebuilds run in parallel, and it brings first-class retries, concurrency limits, and rate limiting.

from celery import Celery

app = Celery("maps", broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")


@app.task(bind=True, max_retries=3, default_retry_delay=60, acks_late=True)
def rebuild_region(self, region_id: str) -> str:
    """Rebuild one region's tiles. Idempotent so retries are safe."""
    try:
        run_region_rebuild(region_id)   # your unit of work
    except TransientError as exc:
        # exponential backoff: 60s, 120s, 240s, then give up
        raise self.retry(exc=exc, countdown=60 * (2 ** self.request.retries))
    return region_id


# Celery beat schedule: fan out all regions every hour
app.conf.beat_schedule = {
    "hourly-fanout": {
        "task": "maps.dispatch_all_regions",
        "schedule": 3600.0,
    }
}


@app.task
def dispatch_all_regions() -> int:
    regions = load_region_ids()
    for rid in regions:
        rebuild_region.delay(rid)   # each region runs on any free worker
    return len(regions)

The max_retries with exponential backoff, acks_late=True so a crashed worker’s task is redelivered, and the fan-out via .delay() are the capabilities that justify Celery’s operational cost. Scale workers horizontally with celery -A maps worker --concurrency=4 per host. The end-to-end Redis-backed task setup is detailed in Running Map-Rebuild Jobs as Celery Tasks with Redis. Do not reach for Celery until the workload actually demands distribution — a broker plus workers is real infrastructure to run, secure, and monitor.

Step 5 — Add Monitoring and Idempotency

A scheduler that runs is not the same as a scheduler you can trust. Two disciplines make the difference at every tier, and they are mandatory once retries or concurrency enter the picture.

Idempotency. A rebuild must be safe to run twice — because retries, overlapping triggers, and manual re-runs all cause it. Write output to a temporary path and atomically rename it into place so readers never see a half-written artifact, and key the work on the input version so a repeat run is a no-op:

import os
import tempfile


def atomic_write_tiles(build_fn, final_path: str) -> None:
    """Build into a temp file, then atomically swap. Safe under retries."""
    dir_ = os.path.dirname(final_path)
    fd, tmp = tempfile.mkstemp(dir=dir_, suffix=".tmp")
    os.close(fd)
    try:
        build_fn(tmp)              # do all work on the temp artifact
        os.replace(tmp, final_path)  # atomic on POSIX; readers see old or new
    except Exception:
        os.unlink(tmp)             # never leave a partial file behind
        raise

Observability. For Celery, run flower (celery -A maps flower) for a live view of queues, worker health, and task history. For cron and APScheduler, structured logs plus a completion heartbeat are enough. Wire these into your Scheduled Map Rebuild Workflows so a missed or failed rebuild raises an alert rather than silently serving stale tiles. Atomic writes also pair naturally with cache-busting once the new artifact lands.

Step 6 — Verify Job Execution

Never assume a schedule fired. Verify with a heartbeat and an output-freshness assertion so a broken scheduler is loud, not silent.

import time
from pathlib import Path

HEARTBEAT = Path("/var/lib/maps/last_rebuild")


def record_heartbeat() -> None:
    """Call at the end of a successful rebuild."""
    HEARTBEAT.write_text(str(int(time.time())))


def assert_fresh(max_age_seconds: int) -> None:
    """Fail loudly if the last successful rebuild is too old."""
    if not HEARTBEAT.exists():
        raise SystemExit("No rebuild heartbeat found — scheduler never ran.")
    age = time.time() - int(HEARTBEAT.read_text())
    if age > max_age_seconds:
        raise SystemExit(f"Stale rebuild: {age:.0f}s old (limit {max_age_seconds}s).")
    print(f"OK: last rebuild {age:.0f}s ago.")

Run assert_fresh from an independent monitor — a separate cron entry, an uptime check, or a health endpoint — so it catches the case where the scheduler itself is down. For Celery, cross-check the heartbeat against Flower: a task that shows as SUCCESS but left no fresh artifact points at a broken write path or a non-idempotent race, not a scheduling problem.

Verification & Smoke-Test

Confirm the scheduler is doing its job before trusting it in production:

  • Trigger fires — force the schedule (run the cron line by hand, call the APScheduler job, or .delay() a Celery task) and confirm it executes end to end.
  • Output is fresh — assert the tile or GeoJSON artifact’s modified time is newer than the trigger time; a run that logs success but leaves a stale file is the most dangerous failure.
  • No overlap — start a long run, fire the next trigger, and confirm the guard (flock, max_instances, or acks_late) prevents a second concurrent execution.
  • Retry works — inject a transient failure and confirm the job retries the expected number of times with backoff, then surfaces the failure instead of retrying forever.
  • Restart survives — restart the host or app and confirm scheduled jobs still exist (persistent jobstore for APScheduler, beat schedule for Celery).

Troubleshooting

Why did my cron job run but produce no updated map?

cron runs with a minimal environment and a bare PATH, so a job that works in your interactive shell frequently fails under cron because it cannot find the interpreter, the virtualenv, or expected environment variables. Use absolute paths to both the Python binary and the script, activate or reference the venv explicitly, export any required variables inside the script, and redirect stdout and stderr to a log file with >> rebuild.log 2>&1. The log is where the real error is; cron’s own mail usually goes nowhere on a headless box.

Why do my APScheduler jobs vanish when the app restarts?

The default MemoryJobStore holds jobs in process memory and loses everything on restart. Configure a persistent jobstore — SQLAlchemyJobStore backed by SQLite or Postgres — so scheduled jobs are serialised and reloaded on startup. Combine it with replace_existing=True on add_job so re-registering the same job on boot is idempotent rather than creating duplicates.

Why do my APScheduler jobs run twice (or N times) under Gunicorn?

Each Gunicorn or Uvicorn worker process starts its own BackgroundScheduler, so the job fires once per worker. Fix it by running the scheduler in a single dedicated process separate from the web workers, gating scheduler startup behind an advisory lock so only one worker owns it, or moving scheduling to Celery beat where exactly one beat process owns the timeline. This is the most common reason teams outgrow APScheduler for a multi-worker deployment.

Why does my Celery task retry forever and pile up in the queue?

An unbounded retry — no max_retries — combined with a non-idempotent task re-enqueues on every failure while side effects accumulate, and the queue grows without bound. Set an explicit max_retries and an exponential countdown, make the rebuild idempotent so a repeat run is harmless, and route tasks that exhaust their retries to a dead-letter queue or an alert instead of silently dropping them. Pair this with acks_late=True so a task is only acknowledged after it finishes, not when it starts.

How do I know a scheduled rebuild actually ran and finished?

Do not trust that a schedule fired — verify it. Write a heartbeat timestamp at the end of every successful rebuild and assert, from an independent monitor, that the output artifact’s modified time is newer than the last trigger. For Celery, expose worker and queue state through Flower or structured task-event logs so a missed, stuck, or silently-failing run is visible. A schedule that “runs” but leaves stale tiles is worse than an obvious crash because nobody notices.

Gotchas & Edge Cases

  • cron’s environment is not your shell’s. Absolute paths, an explicit venv, and exported variables inside the script are mandatory; the number-one cron failure is a job that works by hand and silently does nothing under cron.
  • APScheduler multiplies under multiple workers. Any multi-process server runs one scheduler per worker unless you pin it. Decide up front whether scheduling belongs in the web process at all.
  • A broker is a stateful service, not a library. Redis or RabbitMQ needs its own provisioning, memory limits, persistence policy, and monitoring. Adopting Celery means adopting broker operations too.
  • Non-idempotent tasks and retries are incompatible. The moment you enable retries or allow overlapping runs, a rebuild that is not safe to repeat will corrupt output. Make idempotency a precondition, not an afterthought.
  • Timezones and DST bite scheduled triggers. A CronTrigger at 02:30 local time can run twice or skip on daylight-saving transitions. Schedule in UTC for rebuild jobs unless a human-facing local time is genuinely required.
  • Long jobs silently overlap. If a rebuild can run longer than its interval, an unguarded scheduler starts a second copy on top of the first. Always cap concurrency with flock, max_instances, or a broker-level lock.
  • Event-driven beats the clock for change-driven data. If rebuilds only need to happen when data changes, a scheduler polls needlessly; a webhook trigger is both fresher and cheaper.