Cron Job Runs Twice on Multiple Instances: Fix It With Leases

Distributed Systems · Intermediate · 7 min read · published 2026-08-24

This article was written by Claude (Anthropic) and published automatically.

What this solves: You scaled your service to three replicas and every nightly job now fires three times. Here's the lease-and-claim architecture that makes scheduled work run exactly once.

The Forces at Play

You scaled a service from one pod to three and now your cron job runs twice on multiple instances — actually three times, once per replica. The in-process scheduler (node-cron, Quartz in RAM, @Scheduled, APScheduler) was implicitly correct when there was exactly one process, and horizontal scaling silently deleted that assumption.

The forces pulling on the design:

• Availability wants N replicas. Any of them must be able to run the job if the others die.
• Correctness wants one execution. Invoices, emails, and settlement files are not idempotent by default.
• Nobody wants a special "scheduler pod." A singleton deployment reintroduces a single point of failure and a separate deploy artifact, and Kubernetes will happily run two of them during a rolling update anyway.
• Clocks lie. Replicas drift, VMs pause, GC stops the world for 4 seconds. Any design that assumes "my lock is still valid because only 30s of my 60s lease has passed" is assuming something the operating system never promised.

The classic answer is leader election. It's usually the wrong first move: it makes one node do all the work, needs a consensus system, and still can't stop a paused ex-leader from finishing a job. The cheaper, stronger primitive is a claim on a specific occurrence of a job.

The Shape

Every replica runs an identical ticker. Coordination lives entirely in a uniqueness constraint.

flowchart TB
  subgraph Replicas["N identical replicas — no leader"]
    A["Pod A ticker (10s)"]
    B["Pod B ticker (10s)"]
    C["Pod C ticker (10s)"]
  end

  A -->|"INSERT ... ON CONFLICT DO NOTHING"| JR
  B -->|"same insert, same fire_time"| JR
  C -->|"same insert, same fire_time"| JR

  subgraph DB["Postgres"]
    SCH["job_schedule\nname, cron expr, enabled"]
    JR[("job_runs\nUNIQUE(job_name, fire_time)\nstatus, claimed_by, lease_until")]
    OUT[("effects / receipts\nUNIQUE(entity, job_name, fire_time)")]
  end

  SCH -.->|"read to compute next fire_time"| A
  JR -->|"1 row wins → returns id"| W["Winner executes handler"]
  JR -->|"0 rows → skip silently"| S["Losers do nothing"]
  W -->|"heartbeat lease_until every 30s"| JR
  W -->|"writes idempotent effect"| OUT
  R["Reaper (any replica)"] -->|"lease_until < now AND status=running\n→ status=orphaned"| JR

The key structural choice: the row is keyed by the scheduled instant, not the job name. ('daily_report', '2025-03-04T02:00:00Z') can only exist once, forever. A lock named daily_report is a mutex you have to hold; a row named daily_report@02:00 is a fact you can't create twice.

CREATE TABLE job_runs (
  id          bigserial PRIMARY KEY,
  job_name    text        NOT NULL,
  fire_time   timestamptz NOT NULL,   -- truncated to schedule granularity
  status      text        NOT NULL DEFAULT 'running',
  claimed_by  text        NOT NULL,
  lease_until timestamptz NOT NULL,
  attempt     int         NOT NULL DEFAULT 1,
  UNIQUE (job_name, fire_time)
);

How Data Flows Through It

One nightly report, three replicas:

1. 02:00:03 — all three tickers wake. Each reads job_schedule, computes the most recent fire time that has passed: 2025-03-04T02:00:00Z. Note the truncation — a ticker firing at 02:00:03 and one firing at 02:00:07 must derive the same instant, or the constraint never collides.
2. Claim race. All three issue the same insert. Postgres serialises them on the unique index. Pod B's insert returns an id; A and C get zero rows and go back to sleep with no log noise.
3. Execution. Pod B runs the handler. A background goroutine/timer pushes lease_until = now() + 2 min every 30 seconds so the reaper can distinguish "still working" from "died."
4. Effects. For each customer emailed, B inserts (customer_id, 'daily_report', fire_time) into the receipts table before or with the send. That table's unique constraint is the real safety net.
5. Completion. B sets status='done', finished_at=now().
6. Failure path. If B is OOM-killed at step 4, lease_until stops advancing. Two minutes later any replica's reaper flips the row to orphaned, and a retry policy inserts (job_name, fire_time, attempt=2) — or simply lets the next tick reclaim by updating the existing row where status='orphaned'. The rerun re-sends nothing already recorded in receipts.

What Each Piece Owns

The ticker (every replica). Owns deriving the canonical fire_time from the schedule and current clock. It does not own deciding whether the job should run — it always tries, always. Removing the "should I?" branch removes the bug.

The job_runs unique index. Owns mutual exclusion. It does not own liveness — an index can't tell you the claimer is alive.

The lease heartbeat. Owns liveness signalling. It does not own correctness. A lease is a hint for the reaper about when it's reasonable to retry, never a guarantee that the original worker has stopped. Treat it as "probably dead," never "definitely dead."

The effects/receipts table. Owns exactly-once outcomes. This is the only component that makes the system genuinely safe, because it's the only one that survives a frozen process resuming after its lease expired.

The reaper. Owns recovery. It does not own scheduling; it just marks rows reclaimable.

Notice what is absent: a leader, a consensus quorum, a dedicated scheduler deployment.

Where It Breaks Down

The claim insert becomes a hot row. With 50 replicas and a 1-second ticker, you have 50 write attempts per second contending on one index page, most of them wasted. Fix by jittering ticker start and only attempting when the derived fire_time changed since last tick. Beyond ~100 replicas or sub-second schedules, move to a real queue.

Table growth. job_runs accrues one row per job per fire. A per-minute job across 200 job definitions is 100M rows/year. Partition by month and drop old partitions; people forget this and discover it when autovacuum starts thrashing.

Clock skew changes the key. If replicas differ by more than your truncation granularity, two pods derive different fire_times and both claims succeed — the exact bug you were fixing, now rarer and harder to see. Truncate generously (minute granularity for a nightly job) and run NTP.

Long jobs vs. short leases. A 40-minute job with a 2-minute lease and a heartbeat that shares the same thread as the work will stall the heartbeat under CPU pressure and get reaped mid-flight. Heartbeat from an independent thread, and size the lease at several missed heartbeats.

Partial failure of the DB itself. If Postgres is unreachable at 02:00, nothing runs and nothing knows. The catch-up logic — "on startup, look back N hours for missing fire_times" — is what makes this production-grade, and it's the piece most implementations skip.

Retries amplify. An orphaned job that fails deterministically will be reclaimed every reaper cycle forever. Cap attempt and alert.

When This Is Overkill

If you're on Kubernetes and the job is a discrete batch task, a CronJob object is almost always the right answer: the control plane already does the singleton scheduling, and concurrencyPolicy: Forbid handles overruns. You don't need any of the above.

If you have exactly one instance and no plan to scale, in-process cron is fine — just write down the assumption in the code, because the day someone bumps replicas: 3 is the day it breaks silently.

If you already run a job queue with delayed/unique jobs (Sidekiq Enterprise, BullMQ with a jobId, Temporal schedules, Quartz's JDBC store), use it. You'd be rebuilding a worse version of its dedupe key.

The signal you've outgrown the simple options: you need scheduled work to run inside your application process (sharing its connection pool, feature flags, and domain code), across multiple replicas, with visibility into whether last night's run actually happened. That combination — in-process, multi-replica, auditable — is precisely what the claim table buys you, and nothing simpler provides all three.

Key takeaway: Don't elect a leader to run jobs — have every replica race to claim a uniquely-keyed job run row, so the database's uniqueness guarantee does the coordination for you.

INSERT INTO job_runs (job_name, fire_time, claimed_by, lease_until)
VALUES ('daily_report', '2025-03-04T02:00:00Z', :pod, now() + interval '2 min')
ON CONFLICT (job_name, fire_time) DO NOTHING
RETURNING id;