Consumer groups and rebalancing

A consumer group is how Kafka shares a topic's partitions among a set of consumers so that each record is processed once by the group. It is also where most Kafka operational pain lives, because the sharing is renegotiated whenever the membership changes, and while it is being renegotiated nobody is processing anything. This lesson is the group protocol, how partitions get assigned, what a rebalance costs, and the three settings and one protocol change that keep rebalances rare and short.

#The group protocol

Consumers with the same group.id form a group. One broker acts as the group's coordinator; consumers join it, it chooses a leader among them, and the leader computes an assignment of partitions to members that every member then receives. Each partition is owned by exactly one member; a member may own several; a member with none sits idle. Consumers send heartbeats to the coordinator from a background thread, and the coordinator declares a member dead when heartbeats stop.

The consequence everyone learns the hard way: one partition, one consumer. A topic with six partitions and a group of ten consumers has four consumers doing nothing. Throughput scales by adding partitions, not consumers, and the partition count is a decision made when the topic was created.

#Assignment strategies

partition.assignment.strategy decides how the leader shares the partitions:

#What a rebalance is, and what it costs

A rebalance is the group recomputing the assignment. It happens when a member joins, when a member leaves or is declared dead, and when the subscription or the topic's partition count changes. Under the original eager protocol every member first revokes all its partitions, then the group re-joins, then the new assignment is handed out. Between revoke and re-assign, the whole group processes nothing: a stop-the-world pause of seconds to a minute for a large group, during which lag climbs on every partition.

The rebalance storm is what happens when rebalances trigger rebalances. A deploy restarts twenty consumers one at a time; each departure and each arrival is a rebalance; each rebalance pauses processing; the pause makes a slow consumer exceed its poll interval; the coordinator kicks it out, which is another rebalance. A group can spend minutes doing nothing but rebalancing while lag alerts fire.

#Cooperative rebalancing

CooperativeStickyAssignor (Kafka 2.4) changes the protocol to incremental: a rebalance revokes only the partitions that must move, and the members keeping their partitions keep processing throughout. A new consumer joining a group of ten takes a fair share from the others in two quick rounds, and the other nine never stop. It is the single most effective change for a group that rebalances often, and it requires every member to be on it, so switching a running group is a two-step deploy (add the cooperative assignor alongside the old one, then remove the old one) or a brief full stop.

Kafka 4.0 goes further with the new consumer group protocol (KIP-848): the broker computes assignments itself, members are reconciled individually, and there is no group-wide synchronisation barrier at all. New clients on a 4.0 cluster get it with group.protocol=consumer; older groups keep working on the classic protocol.

#The three timeouts

session.timeout.ms=45000        # heartbeats missing this long → member declared dead (default since 3.0)
heartbeat.interval.ms=3000      # how often the background thread heartbeats; a third of the session timeout
max.poll.interval.ms=300000     # longest gap allowed between two poll() calls → otherwise the member leaves
max.poll.records=500            # records per poll; the amount the processing loop must finish within the interval

Heartbeats prove the process is alive; poll() proves the processing is alive. A consumer that heartbeats happily while its processing thread is stuck in a slow database call for six minutes is removed from the group for exceeding max.poll.interval.ms, its partitions are given to someone else, and when it finally calls poll() it is told it lost them and its uncommitted work is redone elsewhere. That is the most common cause of a rebalance in a healthy cluster, and the fix is either faster processing, fewer records per poll, or, for genuinely long work, handing the record to a worker pool and pausing the partition until it is done.

#Static membership

Every restart of a consumer is a leave and a join, two rebalances, even when the same process comes straight back on the same partitions. group.instance.id gives a member a stable identity: a restarted consumer with the same instance id within session.timeout.ms reclaims its partitions with no rebalance at all. Set it per pod (the pod name, or the StatefulSet ordinal) and a rolling deploy becomes twenty quiet restarts instead of forty rebalances. The trade is that a member that dies for real is not replaced until the session timeout expires, so the timeout is now the failover time, and 45 seconds is about right.

#Sizing the group

• Partitions ≥ consumers, always; extra consumers are idle.
• Consumers per topic = target throughput ÷ one consumer's throughput, then round up, then make sure the partition count allows it.
• One consumer per thread. KafkaConsumer is not thread-safe; Spring's listener container runs concurrency consumers in one process, each on its own thread, each a group member. concurrency above the partition count is idle threads.
• Lag (kafka.consumer.fetch.manager.records.lag.max, or the kafka-consumer-groups tool) is the number that says whether the group keeps up. Alert on it per group; it rises during rebalances and must fall again afterwards.

#Under the hood: the coordinator, the two protocols, and what a heartbeat carries

The group coordinator is the broker that leads the __consumer_offsets partition for hash(group.id) mod 50; a consumer finds it with a FindCoordinator request and talks to it for the group's whole life. Under the classic protocol, joining is two round trips: JoinGroup, in which every member sends its subscription and the coordinator picks one member as leader and returns everyone's subscriptions to it; then SyncGroup, in which the leader sends the assignment it computed (with the configured assignor, on the client) and each member receives its share. The coordinator does not know the assignment logic; it only relays. Heartbeats run on a dedicated background thread every heartbeat.interval.ms, and their response is how the coordinator tells a member that a rebalance is in progress: the next heartbeat returns REBALANCE_IN_PROGRESS, the member's next poll() notices, revokes (all partitions under eager, none yet under cooperative) and re-joins. max.poll.interval.ms is enforced on the client: if the application does not call poll() in time, the heartbeat thread itself sends LeaveGroup, which is why a stuck handler evicts its own consumer.

The cooperative protocol changes only the revocation step, but it changes everything about cost. In the first round, the leader computes the new assignment and, for any partition that must move, assigns it to nobody: the current owner keeps polling until it sees the assignment, then revokes just that partition, commits its offsets, and triggers a second, quick rebalance in which the partition is handed to the new owner. Members whose partitions do not move never stop. Two rounds instead of one, and a group of forty that gains one member sees thirty-nine members keep working. The rule that every member must speak the same protocol exists because an eager member would revoke everything in round one while cooperative members kept theirs, and the assignment the leader computed would be wrong for both.

Kafka 4.0's protocol (KIP-848) moves assignment to the broker. Members send a single periodic ConsumerGroupHeartbeat carrying their subscription and current assignment; the coordinator computes the target assignment server-side and, in each heartbeat response, tells the member which partitions to give up or take, one member at a time as partitions become free. There is no leader, no JoinGroup/SyncGroup barrier, and no group-wide epoch that everyone must reach; a joining member is reconciled without the others noticing. session.timeout.ms and the assignor become broker-side group configs, and client-side custom assignors are replaced by server-side ones (uniform, range).

#Walkthrough: forty rebalances per deploy

A payments consumer with twenty pods on a 40-partition topic took eleven minutes of near-zero throughput on every rolling deploy, and lag alerts fired every time.

spring.kafka.consumer.properties:
  partition.assignment.strategy: org.apache.kafka.clients.consumer.RangeAssignor   # eager
  session.timeout.ms: 10000
  max.poll.interval.ms: 300000

1. Each pod restart was a LeaveGroup (pod stopping) and a JoinGroup (new pod), so two eager rebalances per pod, forty per deploy, each pausing all twenty members for the time it took the slowest to finish its in-flight batch and re-join: five to fifteen seconds.
2. Some members were mid-way through a batch of 500 records with database writes when the rebalance began; under eager they could not finish before revoking, so they committed nothing, the partitions moved, and the next owner reprocessed the batch. Idempotent handlers made that safe and slow.
3. Two members were evicted during the storm: a pause pushed a handler over max.poll.interval.ms, the heartbeat thread sent LeaveGroup, and the group rebalanced again. Forty became forty-four.
4. The changes, in the order deployed: CooperativeStickyAssignor added alongside RangeAssignor in one deploy (members negotiate the common protocol) and RangeAssignor removed in the next; group.instance.id set to the pod name so a restart within session.timeout.ms reclaimed its partitions with no rebalance; session.timeout.ms raised to 45 s, above the pod's restart time; max.poll.records lowered to 100 so a batch finished inside a pause.
5. The next deploy was twenty reclaims and zero rebalances; lag rose by a few hundred records per restarted pod and drained in seconds. The cost was that a pod that crashed for real was not replaced for 45 s, which for this pipeline was acceptable and was written down.

Every restart is two membership changes unless the group is told it is the same member coming back. Static membership makes it a reclaim; cooperative rebalancing makes what remains cheap.

#Try it yourself

#Misconceptions

• "The broker assigns partitions." Under the classic protocol a client-side leader computes the assignment and the coordinator relays it. Only KIP-848 (Kafka 4.0) moves it server-side.
• "Heartbeats prove the consumer is working." They prove the process is up. Progress is proven by poll(); a stuck handler is evicted by its own heartbeat thread.
• "Cooperative rebalancing avoids rebalances." It still rebalances, in two rounds; it avoids stopping members whose partitions do not move.
• "Restarting a consumer is free." Without static membership it is a leave and a join, two rebalances; with it, a reclaim, at the price of slower replacement when it truly dies.
• "Adding consumers adds throughput." Only up to the partition count; beyond that they idle. Per-partition consumer throughput and partition count are the levers.

#Going deeper

• Kafka documentation, consumer configs, and the "Consumer" and "Group membership" sections of the design page.
• KIP-429 (incremental cooperative rebalancing) and KIP-345 (static membership), both short and readable.
• KIP-848 (the next-generation consumer rebalance protocol), for what Kafka 4.0 changes and why.
• Confluent, "Kafka Consumer Group Protocol" and the "Incremental Cooperative Rebalancing" blog post by Sophie Blee-Goldman.
• AbstractCoordinator and ConsumerCoordinator in the Kafka client source.