Kafka's max.poll.interval.ms: The Rebalance Loop That Reprocesses the Same Batch Forever

Kafka · Intermediate · 7 min read · published 2026-08-11

What this solves: Your Kafka consumer group keeps rebalancing every few minutes, commits fail with CommitFailedException, and the same messages get processed over and over while lag climbs. This happens when per-record processing is slow enough that your loop misses the poll deadline.

The Problem

A payments-enrichment consumer runs fine in staging. In production it hits this every few minutes, on every instance:

INFO  Member consumer-orders-3-8f2a sending LeaveGroup request
      because the poll timeout has expired
ERROR Offset commit cannot be completed since the consumer is not
      part of an active group ... you can address this by increasing
      max.poll.interval.ms or reducing max.poll.records

Symptoms: consumer lag climbs steadily to 400k, CPU sits at 15%, and the finance team reports the same order enriched three or four times. The group rebalances roughly every 5 minutes. Nothing crashed. No OOM. No broker errors.

The math: max.poll.records=500 (the default), each record makes one synchronous HTTP lookup averaging 600ms with a p99 of 2s. 500 × 0.6s = 300 seconds — exactly max.poll.interval.ms. So half the batches blow the deadline. And because eviction happens mid-batch, the commit at the end always fails, so the offsets never advance and the next generation of the group fetches the identical 500 records. It's not slow. It's a treadmill.

Why the Obvious Fix Falls Short

The first instinct — and there's a whole generation of StackOverflow answers pointing here — is to tune the timeouts. "The consumer is being kicked out, so raise session.timeout.ms and lower heartbeat.interval.ms."

That does nothing, because since Kafka 0.10.1 heartbeats live on a separate background thread from your processing loop. Your consumer was never failing its heartbeat; the broker knew it was alive the whole time. session.timeout.ms governs liveness. max.poll.interval.ms governs progress. They are two independent failure detectors, and the log message tells you exactly which one fired: "the poll timeout has expired."

The second instinct is the one the error message itself suggests: crank max.poll.interval.ms to 30 minutes. This stops the log spam, and that's the trap. You've now told the coordinator that a consumer that stops calling poll() should be tolerated for 30 minutes. When a pod actually deadlocks or its downstream dependency hangs, its 3 partitions are frozen for half an hour before anyone else can take them — with heartbeats cheerfully flowing the entire time. You traded a loud, self-inflicted problem for a silent, unbounded stall.

The third instinct — commit after every record — reduces the duplicate window but not the deadline. You still don't call poll() for 300 seconds, so you still get evicted; you just get evicted having made partial progress.

The real insight: poll() is not just "fetch messages." It is the consumer's liveness and progress signal, its rebalance-participation callback, and its offset-commit driver, all in one function. Any design where poll() is called rarely is fighting the client.

How It Actually Works

A consumer has two clocks running against it:

• The heartbeat thread pings the group coordinator every heartbeat.interval.ms (3s). Miss session.timeout.ms (45s) worth and you're declared dead.
• The application thread must return to poll() within max.poll.interval.ms (300s). Miss it and the client itself proactively sends LeaveGroup — the broker didn't kick you; your own client resigned on your behalf.

Once that LeaveGroup goes out, your commitSync() at the end of the batch arrives with a stale generation ID and the coordinator rejects it. Offsets stay put. You rejoin, get partitions back, fetch the same offsets, and repeat.

sequenceDiagram
    participant App as App thread
    participant HB as Heartbeat thread
    participant C as Group coordinator
    App->>C: poll() -> 500 records (gen 7)
    Note over App: processing... 600ms x 500
    loop every 3s
        HB->>C: Heartbeat (alive)
        C-->>HB: OK
    end
    Note over HB: t=300s poll deadline blown
    HB->>C: LeaveGroup (poll timeout expired)
    C->>C: Rebalance -> generation 8
    App->>C: commitSync(offset 12500)
    C-->>App: CommitFailedException (stale gen 7)
    App->>C: JoinGroup / SyncGroup
    C-->>App: same partitions, offset 12000
    Note over App,C: identical batch refetched -> duplicates, lag flat

The fix follows from the diagram: shrink the work done between two poll() calls, or decouple processing from polling entirely and use pause()/resume() so you can keep calling poll() cheaply while work happens elsewhere.

Before and After

// BEFORE: one poll() every ~300s. The batch size is set by a default
// nobody chose, and per-record latency is unbounded.
props.put("max.poll.records", 500);          // default
props.put("max.poll.interval.ms", 300_000);  // default

while (running) {
    var records = consumer.poll(Duration.ofSeconds(1));
    for (var r : records) {
        enrich(r);              // ~600ms sync HTTP, p99 2s
    }
    consumer.commitSync();      // often throws CommitFailedException
}

// AFTER: batch sized from measured latency, plus a hard budget so a slow
// downstream can never push us past the deadline mid-batch.
props.put("max.poll.records", 50);           // 50 x 2s p99 = 100s < 150s budget
props.put("max.poll.interval.ms", 300_000);  // unchanged: keep stall detection tight

final Duration BUDGET = Duration.ofMillis(150_000); // half the poll interval

while (running) {
    var records = consumer.poll(Duration.ofSeconds(1));
    long deadline = System.nanoTime() + BUDGET.toNanos();
    Map<TopicPartition, OffsetAndMetadata> done = new HashMap<>();

    for (var r : records) {
        if (System.nanoTime() > deadline) break;   // bail out, commit what's done
        enrich(r);
        done.put(new TopicPartition(r.topic(), r.partition()),
                 new OffsetAndMetadata(r.offset() + 1));
    }
    if (!done.isEmpty()) consumer.commitSync(done); // partial progress is real progress
    // unprocessed records are simply refetched next poll -- offsets guarantee it
}

For genuinely long per-record work (video transcode, LLM calls), go further: hand records to a bounded executor, consumer.pause(assignment) while it's saturated, keep calling poll() on a tight loop to stay alive, and resume() when capacity frees up. Commit only offsets the executor has confirmed.

When NOT to Use This

• Don't shrink max.poll.records reflexively. If your handler is a cheap in-memory transform writing to a batched sink, tiny batches destroy throughput — you lose the amortization of writes and pay per-poll fetch overhead. Measure batch wall time first; if it's 4 seconds, leave it alone.
• If work is inherently minutes-long per message, Kafka's consumer loop is the wrong container. Use the topic as a job pointer: consume the ID, commit immediately, and run the work in a queue/worker system that has its own visibility timeout and retry semantics. Fighting max.poll.interval.ms for a 20-minute job is a losing battle.
• The pause/resume + executor pattern is not free. You've now got ordering, in-flight tracking, and shutdown-drain concerns to get right. For a handler that's 300ms, don't take on that complexity.
• Kafka Streams users: you tune this differently — max.poll.records interacts with the punctuation and commit-interval machinery, so prefer max.poll.records plus commit.interval.ms over hand-rolled loops.

Gotchas

• Cooperative rebalancing hides it, then doesn't. With CooperativeStickyAssignor, rebalances stop being stop-the-world, so the lag graph looks less catastrophic and the incident goes unnoticed for weeks. The duplicate processing is still happening.
• max.poll.interval.ms is now the upper bound on your rolling deploys too. If you raise it to 15 minutes and a pod is SIGKILLed without a clean close(), its partitions sit idle for up to 15 minutes. Always call consumer.close() (or use a shutdown hook) so it sends an explicit LeaveGroup.
• Averages lie. Size the batch from p99, not mean, latency. A downstream that degrades from 600ms to 2s — a totally normal Tuesday — is what flips a comfortable config into a rebalance loop.
• CommitFailedException is not retryable. Catching it and calling commitSync() again just throws again. The only correct response is to let the rebalance complete and reprocess idempotently.
• Idempotency is mandatory, not optional. Any rebalance — a deploy, a scale-up, a network blip — replays uncommitted records. If your consumer isn't safe to run twice on the same offset, you have a data-corruption bug waiting for the next Kubernetes node drain.
• Watch the right metric. records-lag-max tells you you're behind; poll-idle-ratio-avg near 0 and a rising rebalance-rate-per-hour tell you why. Alert on rebalance rate — a healthy group's should be near zero between deploys.

Key takeaway: Heartbeats keep you in the group but only poll() proves you're making progress — size max.poll.records so one batch always finishes well inside max.poll.interval.ms, or hand work off and use pause()/resume().

long t0 = System.nanoTime();
var records = consumer.poll(Duration.ofMillis(500));
process(records);
log.info("batch={} ms={}", records.count(), (System.nanoTime()-t0)/1_000_000);