Running Map-Rebuild Jobs as Celery Tasks with Redis

Part of the Choosing a Scheduler: Celery vs APScheduler vs cron guide.

Operative rule: with acks_late a Redis queue delivers each rebuild at least once, so the task must be idempotent — guard it with a rebuild key so a redelivered message becomes a safe no-op, never a second rebuild.

How Celery + Redis Rebuilds Work

Celery splits a job into three moving parts: a broker that holds a queue of messages, one or more workers that pull messages and execute the matching task, and an optional result backend that stores each task’s state and return value. Redis can play both broker and result backend, which is why it is the pragmatic default for a single geo-dashboard: one process to run, one connection string to configure. You decorate a Python function with @app.task, call rebuild_tiles.delay(dataset) from a web handler or from celery beat, and a worker somewhere picks it up and regenerates the artifact — off the request thread, on its own schedule, and scalable by simply starting more workers.

The subtlety that decides whether this is reliable is delivery semantics. By default Celery acknowledges a message the moment a worker receives it; if that worker then crashes, the task is lost. Setting acks_late=True moves the acknowledgement to after the task returns, so a crash mid-rebuild causes the broker to redeliver the message and the rebuild runs again. That converts “lost on crash” into “runs at least once” — strictly safer, but only if the task tolerates running twice. The tool for that is an idempotency key: a deterministic token derived from the rebuild’s inputs, held in Redis, that lets the second execution detect “this exact rebuild already happened” and exit cleanly. This is the same discipline that incremental data processing relies on to make repeated runs cheap. For a single box that never needs more than one worker, an in-process scheduler is often simpler — see scheduling map rebuilds with APScheduler in Flask — and choosing cron vs APScheduler for single-server dashboards covers the even-simpler end.

Production-Ready Implementation

The module wires Redis as broker and result backend, schedules a nightly rebuild with celery beat, and makes the task safe under redelivery: acks_late plus a SETNX-based idempotency lock keyed on the rebuild inputs, plus bounded retries with exponential backoff.

from __future__ import annotations

import hashlib
import os

import redis
from celery import Celery
from celery.schedules import crontab

REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")

app = Celery("map_rebuilds", broker=REDIS_URL, backend=REDIS_URL)
app.conf.update(
    task_acks_late=True,               # ack AFTER completion -> at-least-once
    task_reject_on_worker_lost=True,   # requeue if the worker dies mid-task
    worker_prefetch_multiplier=1,      # long jobs: fetch one at a time
    task_default_retry_delay=30,
    result_expires=3600,
)

# A separate connection for the idempotency lock and state flags.
_rdb = redis.Redis.from_url(REDIS_URL)


def _rebuild_key(dataset: str, version: str) -> str:
    """Deterministic idempotency key: same inputs -> same key."""
    digest = hashlib.sha256(f"{dataset}:{version}".encode()).hexdigest()[:16]
    return f"rebuild:done:{digest}"


@app.task(
    bind=True,
    max_retries=3,
    acks_late=True,
    autoretry_for=(OSError,),          # transient IO -> retry
    retry_backoff=True,                # 30s, 60s, 120s ...
    retry_backoff_max=600,
)
def rebuild_tiles(self, dataset: str, version: str) -> dict[str, str]:
    """Idempotent tile rebuild. Safe to run more than once for one schedule."""
    key = _rebuild_key(dataset, version)

    # SET NX EX: claim the key only if it does not already exist. A redelivered
    # copy of the same task finds the key present and exits without rebuilding.
    claimed = _rdb.set(key, self.request.id, nx=True, ex=86_400)
    if not claimed:
        return {"status": "skipped", "reason": "already rebuilt", "dataset": dataset}

    try:
        # ... run tippecanoe / regenerate GeoJSON / swap the served artifact ...
        # Write to a temp path, then atomically rename into place on success.
        return {"status": "rebuilt", "dataset": dataset, "version": version}
    except Exception:
        # Release the claim so a retry can genuinely re-run the work.
        _rdb.delete(key)
        raise  # autoretry_for / max_retries handle the backoff


# celery beat: one scheduler enqueues the nightly rebuild.
app.conf.beat_schedule = {
    "nightly-tile-rebuild": {
        "task": "map_rebuilds.rebuild_tiles",
        "schedule": crontab(hour=2, minute=0),  # 02:00 in the worker timezone
        "args": ("parcels", "2026-07-10"),
    },
}

Run the two processes separately — one beat, one or more workers:

# Exactly one beat process (duplicate beats = duplicate schedule entries).
celery -A tasks beat --loglevel=info

# Scale workers for concurrency; each runs the idempotent task safely.
celery -A tasks worker --concurrency=4 --loglevel=info

Alternative Variants

Enqueue on demand from a web handler

Beyond the scheduled run, a webhook or admin action can enqueue the same task with .delay(); the idempotency key means a duplicate enqueue for the same version collapses to one rebuild.

from tasks import rebuild_tiles

# Returns immediately; the worker does the heavy lifting off-thread.
async_result = rebuild_tiles.delay("parcels", "2026-07-10")
print(async_result.id)  # track state via the Redis result backend

Reliability setting reference

Verification Steps

• At-least-once proven — kill a worker with SIGKILL mid-rebuild and confirm the broker redelivers the task and it completes on the second worker.
• Idempotency holds — enqueue the same (dataset, version) twice in quick succession and confirm the second returns {"status": "skipped"} and writes nothing.
• Retry backoff — raise OSError on the first attempt in a staging build and confirm the task retries with 30s, 60s, 120s spacing, then succeeds.
• Single beat — confirm only one celery beat process is running; two beats produce two nightly enqueues.
• Key release on failure — force an exception, confirm the idempotency key is deleted so a genuine retry re-runs the work rather than skipping.

Common Errors & Fixes

The rebuild runs twice for one nightly schedule

Either two celery beat processes are enqueuing the schedule, or acks_late redelivered a task after a worker restart and the task is not idempotent. Fix: run exactly one beat, and keep the SET NX EX idempotency guard so a redelivered message keyed on the same inputs exits as a no-op instead of rebuilding again.

Task never retries and is marked FAILURE immediately

The raised exception type is not in autoretry_for, so Celery treats it as terminal. Fix: either add the exception class to autoretry_for, or call self.retry(exc=err) explicitly inside an except block. Keep max_retries bounded so a permanently broken dataset does not retry forever.

WorkerLostError and the task silently disappears

A worker was killed (often the OOM killer on a memory-heavy tile build) before acknowledging. Without task_reject_on_worker_lost the message is discarded. Fix: set task_reject_on_worker_lost=True so the broker requeues the task, and lower worker_prefetch_multiplier to 1 so a dying worker is not holding several unacked rebuilds.

Redis fills up with stale result and lock keys

Task results and idempotency keys accumulate without expiry. Fix: set result_expires on the app and always pass a TTL (ex=) when writing the idempotency key, so both classes of key self-evict rather than growing the Redis memory footprint unbounded.

Related

• Choosing a Scheduler: Celery vs APScheduler vs cron — parent guide weighing a distributed queue against simpler in-process options
• Scheduling Map Rebuilds with APScheduler in Flask — the in-process alternative when one box and one worker suffice
• Choosing cron vs APScheduler for Single-Server Dashboards — the even-simpler end of the spectrum for a single server
• Incremental Data Processing — making each rebuild cheap so at-least-once retries stay affordable