Retry Storms: Why Exponential Backoff Without Jitter Still Melts Your Service

Resilience · Intermediate · 6 min read · published 2026-08-10

What this solves: Your service already retries with exponential backoff, yet a 2-second blip turns into a 5-minute outage with traffic arriving in synchronized spikes. This covers why aligned retries amplify load and how jitter plus a retry budget breaks the cycle.

The Problem

Your checkout service depends on an inventory API. At 14:02 the inventory database does a 3-second failover. Nothing dramatic — 3 seconds.

By 14:03 the inventory API is at 100% CPU and returning 503s. It stays down for 11 minutes. Upstream user traffic never changed: still ~2,000 rps.

The request-rate graph on the inventory side is the giveaway. It isn't a smooth ramp. It's a comb: a spike to 14,000 rps, silence, a spike, silence — the gaps doubling each time (1s, 2s, 4s, 8s). Every one of your 6,000 checkout pods retried with textbook exponential backoff, and every one of them started its clock within the same 3-second failure window. So they all came back together, three more times, each time hitting a service that hadn't finished draining the previous wave.

Backoff was implemented correctly. It still caused the outage.

Why the Obvious Fix Falls Short

The reflex is "our backoff isn't aggressive enough — bump base delay from 1s to 5s, cap at 60s."

That doesn't fix anything, because the problem isn't the magnitude of the delay, it's the correlation between clients. Exponential backoff is a single-client optimization: it reduces how often one caller hammers a dying dependency. It says nothing about the aggregate arrival process. If N clients fail at the same instant and all compute base * 2^attempt, they wake at the same instant. Making the delay 5s just relocates the same 14,000-rps spike to a later second — and now with a 60s cap, recovery is slower because the herd's period gets longer while its amplitude stays identical.

The second reflex — "retry more times, it'll eventually get through" — is actively harmful. Retries multiply load precisely when capacity is lowest. And if you retry at multiple layers (SDK 3x inside gateway 3x inside client 3x), the fan-out is multiplicative: 27 requests per user action. A dependency running at 60% utilization can't absorb even a 3x multiplier, let alone 27x.

The third reflex, longer timeouts, has its own trap: if your timeout is below the dependency's recovered p99, you cancel requests that were about to succeed and immediately replace them with fresh ones. You spend all your capacity on work you throw away.

How It Actually Works

Three independent mechanisms, and you need all three.

1. Full jitter turns a deterministic wake-up time into a uniform random draw: sleep = random(0, min(cap, base * 2^attempt)). This converts a spike of height N into a roughly flat arrival rate of N/window. AWS's own measurements found full jitter beats "equal jitter" and "decorrelated jitter" for total work done under contention in most regimes — and it's the simplest to reason about.

2. A retry budget bounds retries as a fraction of successful traffic (e.g. token bucket allowing retries at 10% of the success rate). When the dependency is mostly healthy, retries are nearly free and you get their full benefit. When it's mostly failing, the budget empties and you fail fast — exactly the moment retries stop helping.

3. Retry at one layer only, driven by a propagated deadline. Inner layers return the error; the layer that owns the user-visible deadline decides whether there's time to try again.

flowchart TD
    A[Dependency blips for 3s] --> B[6000 clients fail simultaneously]
    B --> C{Backoff strategy}
    C -->|Fixed exponential| D["All wake at t=1s\nspike 14k rps"]
    D --> E[Overload -> new failures]
    E --> F["All wake at t=3s\nsame spike, doubled period"]
    F --> E
    C -->|Full jitter| G["Wake times uniform in 0..1s\n~ flat 2.5k rps"]
    G --> H{Retry budget<br/>tokens left?}
    H -->|Yes, error rate low| I[Retry succeeds]
    H -->|No, error rate high| J[Fail fast, shed load]
    J --> K[Dependency drains queue and recovers]
    I --> K

The feedback loop is the whole story: unjittered retries create the overload that creates the next round of retries. Jitter breaks the amplitude; the budget breaks the loop.

Before and After

# BEFORE: correct exponential backoff, still causes a synchronized herd
import time, requests

def fetch_inventory(sku, attempts=5):
    for i in range(attempts):
        try:
            r = requests.get(f"{URL}/{sku}", timeout=2)
            if r.status_code < 500:
                return r.json()
        except requests.RequestException:
            pass
        time.sleep(min(30, 1 * (2 ** i)))  # deterministic: every client
                                           # wakes at 1, 2, 4, 8, 16s
    raise InventoryUnavailable(sku)

# AFTER: full jitter + retry budget + retry only on retryable failures
import random, time, requests

budget = TokenBucket(rate_ratio=0.10, min_rate=3)  # retries <= 10% of successes

def fetch_inventory(sku, attempts=4, deadline=None):
    for i in range(attempts):
        try:
            r = requests.get(f"{URL}/{sku}", timeout=2)
            if r.status_code < 500:
                budget.record_success()
                return r.json()          # includes 4xx: do NOT retry those
            retryable = True
        except requests.RequestException:
            retryable = True

        if i == attempts - 1 or not budget.try_consume():
            break                        # shed load when dep is mostly failing
        sleep = random.uniform(0, min(30, 1 * (2 ** i)))   # full jitter
        if deadline and time.monotonic() + sleep > deadline:
            break                        # no time left; don't waste a retry
        time.sleep(sleep)
    raise InventoryUnavailable(sku)

When NOT to Use This

• Non-idempotent writes without an idempotency key. Retrying POST /charge on a timeout can double-charge; the timeout tells you nothing about whether the server applied it. Add an idempotency key first, or don't retry.
• Single-caller batch jobs. One nightly ETL process talking to one warehouse has no herd. Plain backoff is fine; jitter adds nothing but nondeterministic test runs.
• Server-directed backoff. If the dependency returns Retry-After or gRPC RetryInfo, honor it — it knows its recovery state better than your client-side heuristic. Still add a small random offset on top.
• Deep queues instead. For work that doesn't need a synchronous response, a durable queue with a dead-letter path beats in-process retries entirely: it survives your pod restarting, and the consumer controls the drain rate.
• When the real fix is capacity or a circuit breaker. Retries can't rescue a dependency that's structurally undersized. If error rate stays above ~50%, you want a breaker that stops traffic, not smarter retries.

Gotchas

• random.uniform(0, delay) with a seeded or per-process-forked RNG. Fork after seeding and every worker draws the identical jitter sequence. You've reinvented the herd. Verify your RNG is seeded per process.
• Kubernetes makes correlation worse. A rolling deploy starts hundreds of pods within seconds; their health checks, config fetches, and cache warms all align. Jitter your startup work too, not just retries.
• Nested retries hide in SDKs. The AWS SDK, gRPC channel config, Envoy route policy, and your own wrapper each retry by default. Audit them; the multiplier is silent until it isn't.
• Jitter breaks flaky-test debugging. Inject the sleep function so tests can make it deterministic, and log the actual sleep duration so production timelines are reconstructable.
• Retry budgets need a floor. A pure percentage budget starves at low traffic — with 2 rps you'd allow 0.2 retries/s. Add a min_rate so genuinely rare requests still get one retry.
• Timeout below recovered p99 is a load amplifier. Measure the dependency's p99 after recovery, not during steady state, and set your per-attempt timeout above it. Otherwise every retry cancels work that was about to complete.

Key takeaway: Exponential backoff only spreads retries in time for a single client; add full jitter and a client-side retry budget so 10,000 clients don't all come back on the same tick.

sleep = random.uniform(0, min(cap, base * 2 ** attempt))
if budget.tokens_available():  # e.g. retries <= 10% of successes
    retry()
else:
    fail_fast()