Commit Cannot Be Completed Since the Group Has Already Rebalanced: Fix the Kafka Rebalance Loop

Kafka · Intermediate · 7 min read · published

This article was written by Claude (Anthropic) and published automatically. Published by Osama Fayyaz.

What this solves: Your Kafka consumer keeps getting kicked out of its group, fails to commit, and reprocesses the same records in a loop. Here is why it happens and how to stop it.

The Problem

You see CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. Usually it shows up only under load. The consumer worked fine when each record took 20ms. Then a downstream API got slower and each record started taking 800ms.

The arithmetic is simple:

Around record 375, the client decides your application is stuck and leaves the group. The group rebalances, and another consumer picks up the partition from the last committed offset. When your consumer finally finishes and calls commitSync(), the broker rejects it. The new owner then reprocesses the same 375 records, hits the same slow API, gets evicted the same way, and the loop repeats.

What you observe:

In librdkafka-based clients (confluent-kafka-python, Go, .NET) the same failure reads Application maximum poll interval (300000ms) exceeded by 112ms.

Why the Obvious Fix Falls Short

Raising session.timeout.ms is the most common first move, and it does nothing here. Since Kafka 0.10.1 (KIP-62), there are two separate liveness checks:

Your heartbeats are fine. It is the poll timer that expires.

Raising max.poll.interval.ms to an hour or to Integer.MAX_VALUE makes the error go away, and that is the trap. It causes two new problems:

You have traded a loud failure for a quiet one.

How It Actually Works

The poll timer resets only when your thread calls poll(). Everything you do between two poll() calls counts against the budget. That includes processing, database writes, and commitSync().

sequenceDiagram
    participant App as App thread
    participant HB as Heartbeat thread
    participant GC as Group coordinator
    App->>GC: poll() returns 500 records (poll timer starts)
    loop every 3s
        HB->>GC: heartbeat (session OK)
    end
    App->>App: process records at 0.8s each
    Note over App: 300s elapsed, poll timer expires
    HB->>GC: LeaveGroup (client gives up proactively)
    GC->>GC: rebalance, partition reassigned to consumer B
    App->>GC: commitSync(offset 500)
    GC-->>App: CommitFailedException (generation is stale)
    Note over GC: consumer B restarts at the old committed offset, so records are duplicated

The rule to work from:

worst-case per-record time × max.poll.records < max.poll.interval.ms

Leave at least 2× headroom. If you cannot bound the per-record time, stop processing inline. Hand records to workers, pause() the partitions, and keep calling poll(). While partitions are paused, poll() returns no records but still resets the timer.

Before and After

Before: an unbounded batch with blocking work inline.

Properties p = new Properties();
// max.poll.records defaults to 500; max.poll.interval.ms defaults to 300000
p.put("max.poll.interval.ms", "3600000"); // "fix": hides hangs, slows every rebalance

while (running) {
  ConsumerRecords<String, Order> recs = consumer.poll(Duration.ofMillis(500));
  for (var r : recs) {
    enrich(r.value()); // 0.8s HTTP call, x500 = 400s between polls
  }
  consumer.commitSync(); // fails once the group has rebalanced
}

After: a bounded batch, a time budget, cooperative rebalancing, and pause/resume for slow work.

p.put("max.poll.records", "50");          // CHANGED: 50 x 0.8s = 40s, well under 300s
p.put("max.poll.interval.ms", "300000");  // CHANGED: back to the default, so hangs are still detected
p.put("partition.assignment.strategy",
      CooperativeStickyAssignor.class.getName()); // one eviction no longer stops the whole group

while (running) {
  var recs = consumer.poll(Duration.ofMillis(500));
  if (!recs.isEmpty()) {
    consumer.pause(consumer.assignment());     // keep polling without fetching more
    inFlight = pool.submit(() -> recs.forEach(r -> enrich(r.value())));
  }
  if (inFlight != null && inFlight.isDone()) {
    inFlight.get();                            // surface worker exceptions
    consumer.commitSync();                     // commit on the consumer thread only
    consumer.resume(consumer.paused());
    inFlight = null;
  }
}

When NOT to Use This

Gotchas

Key takeaway: Size max.poll.records so a full batch always finishes well inside max.poll.interval.ms, and use pause/resume for slow work, instead of raising the interval until the symptom disappears.

Real-world challenge

An order-enrichment service runs 6 consumer instances on a 12-partition topic. After a downstream geocoding API added rate limiting, consumer lag climbed from near zero to 2 million. The logs show `CommitFailedException` every few minutes, followed by `Revoking previously assigned partitions` on all six instances. The same order IDs show up processed 3–4 times in the audit table. CPU usage is low everywhere. What is going on, and how do you fix it?

Diagnosis

  1. Rate limiting pushed each geocoding call from ~100ms to ~1s. At the default max.poll.records=500, one batch now takes ~500s. That exceeds the default max.poll.interval.ms of 300s.
  2. The poll timer expires and the client proactively leaves the group. That triggers an eager rebalance, which revokes partitions from all six instances. This explains the group-wide Revoking lines.
  3. The slow instance finishes its batch and calls commitSync(). It is no longer a member, so the commit is rejected. Another member then re-reads from the last committed offset, which produces the duplicates.
  4. Low CPU is the giveaway. The consumers are blocked on I/O, not on compute.

Fix

max.poll.records=50
max.poll.interval.ms=300000
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor