Scheduling Map Rebuilds with APScheduler in Flask

Part of the Scheduled Map Rebuild Workflows guide.

Operative rule: run exactly one BackgroundScheduler instance for the whole deployment — gate its startup so that multiple Gunicorn workers or the Werkzeug reloader can never schedule the same rebuild job twice.

How APScheduler Inside Flask Works

APScheduler runs a scheduler object inside your Python process rather than as a separate daemon. A BackgroundScheduler spins up its own thread pool, wakes on a timer, and fires any job whose trigger is due — all without blocking Flask’s request handlers. For a geo-dashboard this means the same process that serves tiles can also regenerate them: a CronTrigger fires a rebuild_tiles() function at 02:00, the function writes fresh .mbtiles or GeoJSON to disk, and the next request serves the updated artifact. Because the scheduler shares the interpreter, the job has direct access to your app config, database session, and the same file paths the request handlers use — the core advantage over an external cron entry, which runs in a bare shell with no application context.

The catch is process multiplicity. Production Flask runs under a WSGI server such as Gunicorn with several worker processes, and each worker imports your app module top to bottom. If scheduler startup lives at import time, every worker starts its own scheduler and every rebuild fires once per worker. The fix is to make scheduler ownership singular. A persistent jobstore — a SQLAlchemyJobStore pointing at the dashboard’s own database — records job definitions and last-run timestamps so the scheduler survives restarts and can honour misfire_grace_time, catching up a rebuild that was missed while the process was down. This page implements the single-owner pattern; when you later outgrow one box, the scheduler selection guide covers moving to a distributed queue, and choosing cron vs APScheduler for single-server dashboards covers when to stay simpler.

Single-owner APScheduler under multiple Gunicorn workers Three Gunicorn workers each import the app. All three attempt to acquire a single lock. Only worker 1 wins, starts the BackgroundScheduler, and fires the rebuild job on its CronTrigger. Workers 2 and 3 skip scheduler startup. Worker 1 imports app Worker 2 imports app Worker 3 imports app Single lock file / advisory lock win skip BackgroundScheduler one instance only rebuild_tiles() CronTrigger 02:00

Production-Ready Implementation

The module below is a complete Flask app factory. Scheduler startup is guarded so that only one process ever owns the jobs: a POSIX file lock (fcntl.flock) that a single worker acquires and holds for the process lifetime. Jobs live in a SQLAlchemyJobStore so they persist across restarts, and misfire_grace_time lets a rebuild that was due during a deploy still run once the process comes back.

from __future__ import annotations

import atexit
import fcntl
import os
from pathlib import Path

from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.cron import CronTrigger
from flask import Flask

# A lock file living outside the code tree. The first process to flock() it
# becomes the sole scheduler owner; the file handle is kept open for the
# process lifetime so the lock is never released early.
_LOCK_PATH = Path(os.getenv("SCHEDULER_LOCK", "/tmp/map_rebuild.lock"))
_lock_handle = None  # module-level so the fd is not garbage-collected


def _acquire_scheduler_lock() -> bool:
    """Return True if THIS process won the exclusive scheduler lock."""
    global _lock_handle
    _lock_handle = open(_LOCK_PATH, "w")
    try:
        fcntl.flock(_lock_handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except OSError:
        _lock_handle.close()
        _lock_handle = None
        return False
    return True


def rebuild_tiles(app: Flask) -> None:
    """Regenerate map artifacts. Runs inside an app context so it can read
    config and reuse the same database as the request handlers."""
    with app.app_context():
        output_dir = Path(app.config["TILE_OUTPUT_DIR"])
        output_dir.mkdir(parents=True, exist_ok=True)
        # ... run tippecanoe / write GeoJSON / swap the served artifact ...
        app.logger.info("Tile rebuild complete -> %s", output_dir)


def _start_scheduler(app: Flask) -> None:
    jobstores = {
        "default": SQLAlchemyJobStore(url=app.config["SCHEDULER_DB_URL"]),
    }
    scheduler = BackgroundScheduler(
        jobstores=jobstores,
        timezone="UTC",
        job_defaults={
            "coalesce": True,          # collapse missed runs into one
            "max_instances": 1,        # never overlap a rebuild with itself
            "misfire_grace_time": 3600,  # still run if <=1h late (e.g. after deploy)
        },
    )
    scheduler.add_job(
        func=rebuild_tiles,
        args=[app],
        trigger=CronTrigger(hour=2, minute=0),  # nightly 02:00 UTC
        id="nightly_tile_rebuild",
        replace_existing=True,  # idempotent registration across restarts
    )
    scheduler.start()
    atexit.register(lambda: scheduler.shutdown(wait=False))
    app.logger.info("APScheduler started; this process owns the rebuild jobs.")


def create_app() -> Flask:
    app = Flask(__name__)
    app.config.update(
        TILE_OUTPUT_DIR=os.getenv("TILE_OUTPUT_DIR", "dist/tiles"),
        SCHEDULER_DB_URL=os.getenv("SCHEDULER_DB_URL", "sqlite:///jobs.sqlite"),
    )

    # Guard 1: under `flask run` the Werkzeug reloader forks a parent watcher
    # and a child; only the child sets WERKZEUG_RUN_MAIN. Skip the parent.
    in_reloader_parent = (
        app.debug and os.getenv("WERKZEUG_RUN_MAIN") != "true"
    )
    # Guard 2: under Gunicorn's multiple workers, only the lock winner starts.
    if not in_reloader_parent and _acquire_scheduler_lock():
        _start_scheduler(app)

    @app.get("/healthz")
    def healthz() -> dict[str, bool]:
        return {"scheduler_owner": _lock_handle is not None}

    return app


app = create_app()

Alternative Variants

IntervalTrigger instead of CronTrigger

When “every 15 minutes” is more natural than a wall-clock time — for a dashboard that ingests near-real-time feeds — swap the trigger. IntervalTrigger measures from the last run rather than an absolute clock, which pairs well with coalesce=True to avoid a thundering herd after downtime.

from apscheduler.triggers.interval import IntervalTrigger

scheduler.add_job(
    func=rebuild_tiles,
    args=[app],
    trigger=IntervalTrigger(minutes=15),
    id="frequent_tile_rebuild",
    replace_existing=True,
    max_instances=1,   # a long rebuild must not stack on the next tick
)

Guard strategy reference

Deployment Duplication source Single-owner guard
flask run (debug) reloader parent + child both import check WERKZEUG_RUN_MAIN == "true"
Gunicorn, N sync workers each worker imports the app fcntl.flock on one lock file
Gunicorn, --preload master imports once, then forks start scheduler in a post_fork hook for one worker
Multiple hosts / containers no shared filesystem lock PostgreSQL advisory lock (pg_try_advisory_lock)

For anything beyond one host, the file lock no longer coordinates and you should move the job off the web process entirely — see running map-rebuild jobs as Celery tasks with Redis.

Verification Steps

  • Single owner confirmed — curl /healthz against each worker (via repeated requests) and confirm only one process ever reports "scheduler_owner": true. With four workers you must never see the flag true on more than one.
  • No double firing — set the trigger to IntervalTrigger(seconds=30) in a staging run and grep "Tile rebuild complete" app.log. Exactly one line should appear per 30-second window, not one per worker.
  • Persistence across restart — start the app, note the job in the apscheduler_jobs table, restart, and confirm replace_existing=True did not create a duplicate row.
  • Misfire catch-up — stop the process just before a scheduled run, restart within the grace window, and confirm the rebuild fires once on startup rather than being silently skipped.
  • Reloader safety — run flask run --debug, edit a file to trigger a reload, and confirm the log shows a single “APScheduler started” line, not two.

Common Errors & Fixes

Rebuild fires once per Gunicorn worker

The scheduler is being started at import time in every worker. Each worker’s BackgroundScheduler sees the same CronTrigger and fires independently, so a four-worker deployment runs four rebuilds that race on the output directory. Fix: gate startup behind a single exclusive lock as shown — fcntl.flock for one host, or a PostgreSQL advisory lock across hosts — so only one process calls scheduler.start().

Scheduler starts twice under flask run

The Werkzeug reloader runs two processes: a parent that watches for file changes and a child that actually serves. Both import your module, so scheduler startup runs twice in development. Only the child sets WERKZEUG_RUN_MAIN=true. Fix: skip startup when app.debug is set and WERKZEUG_RUN_MAIN is not "true", or pass use_reloader=False to app.run().

Job "nightly_tile_rebuild" missed by ... skipped

The process was down when the trigger was due, and with the default misfire_grace_time of None for that job APScheduler declined to run it late. Fix: set a misfire_grace_time (e.g. 3600) in job_defaults so a job that comes due during a deploy still runs once the scheduler is back, and set coalesce=True so several missed runs collapse into a single execution.

Rebuilds overlap and corrupt the output directory

A rebuild is taking longer than the interval between triggers, so a second run starts while the first is still writing. Fix: set max_instances=1 on the job so APScheduler refuses to launch a second copy while one is running, and write to a temporary directory that you atomically rename into place only on success.