What Triggers a Kafka Consumer Group Rebalance?

The short version

A rebalance is the process by which a Kafka consumer group redistributes topic partitions across its members. It happens when one of three things changes:

1. Group membership — a consumer joins, shuts down cleanly, misses heartbeats for longer than session.timeout.ms, or fails to call poll() within max.poll.interval.ms.
2. The subscription — a consumer calls subscribe() with a different set of topics.
3. The metadata the subscription resolves to — partitions are added to a subscribed topic, a subscribed topic is deleted, or a newly created topic starts matching a subscribe(Pattern) regex.

Everything else you will read about rebalances — coordinator failover, rolling deploys, GC pauses — reduces to one of those three. The rest of this article walks each trigger, gives the config that governs it, and separates the two failure detectors that people most often confuse.

Who actually runs a rebalance

A consumer group is a set of clients sharing a group.id. Its state lives on one broker, the group coordinator for that group. Which broker? Kafka hashes the group.id to a partition of the internal __consumer_offsets topic, and the leader of that partition is the coordinator. __consumer_offsets has 50 partitions by default (offsets.topic.num.partitions, see the broker configs), so groups spread across the cluster.

In the classic protocol — everything before KIP-848 — the coordinator does not compute the assignment. It picks one member as the group leader, ships that member the full list of subscriptions in the JoinGroup response, and the leader runs a ConsumerPartitionAssignor locally. The result comes back to the coordinator in a SyncGroup request, which fans it out to everyone. The broker is a rendezvous point and a referee; the arithmetic is client-side. This is documented in KIP-848, which changes it.

The group moves through a state machine: Empty → PreparingRebalance → CompletingRebalance → Stable. Each completed rebalance bumps a generation id, a monotonically increasing counter stamped on every request. If a member is still working with the old generation — say it was in a GC pause while the group moved on — its next request is rejected with ILLEGAL_GENERATION, and it must rejoin. Separately, while a rebalance is in progress the coordinator answers a current member's in-flight requests with REBALANCE_IN_PROGRESS, which is how a live member learns it has to rejoin the new generation.

Trigger 1: membership changes

Any JoinGroup that changes what the group has to agree on moves a Stable group to PreparingRebalance — whether it comes from a member the coordinator has never seen or from an existing member rejoining, for instance with a changed subscription (Trigger 3 below). Start a second consumer, and the group rebalances. Start a twentieth, and it rebalances again.

This is why a naive rolling deploy is expensive. Twenty instances, restarted one at a time, means twenty leave events and twenty join events. Under the eager protocol every one of those pauses the whole group.

Shutdown matters too. Calling KafkaConsumer.close() sends an explicit LeaveGroup request, so the coordinator rebalances immediately rather than waiting for the session timeout to expire. That is usually what you want — a clean shutdown that hands partitions over in seconds instead of leaving them stranded — but for a dynamic member under the classic protocol it does mean a graceful restart costs you two rebalances, not one (with static membership, discussed below, a bounce inside session.timeout.ms costs none). Kill the process without closing, and you trade the immediate rebalance for up to session.timeout.ms of stalled partitions.

One broker config softens the startup case: group.initial.rebalance.delay.ms, default 3000 (3 seconds). When a group is empty and the first member joins, the coordinator deliberately waits, letting more members arrive so a batch of starting consumers is absorbed into a single rebalance. Note the constraint — it only applies to the empty group case. It does nothing for a rolling bounce of a running group.

Trigger 2: the coordinator decides a member is dead

Here is the part that trips people up. There are two independent failure detectors, they fire for different reasons, and they have different fixes.

Since KIP-62 (Apache Kafka 0.10.1.0), the consumer sends heartbeats from a dedicated background thread. Before that, heartbeats piggybacked on poll(), so slow processing and a dead process looked identical to the broker. Splitting them apart created the two timers below.

	Liveness detector	Progress detector
Config	session.timeout.ms	max.poll.interval.ms
Default	45000 (45 s)	300000 (5 min)
Measures	Gap between heartbeats	Gap between poll() calls
Who notices	The coordinator	The consumer itself
Who acts	Coordinator evicts the member	Consumer sends LeaveGroup
Typical cause	GC pause, network partition, hung JVM	Slow business logic, blocking downstream call, oversized batch

The liveness path. The background thread sends a Heartbeat request every heartbeat.interval.ms, default 3000 (3 s). If the coordinator receives none within session.timeout.ms, it removes the member and rebalances. The consumer config docs state heartbeat.interval.ms must be lower than session.timeout.ms and typically no higher than a third of it — you want several heartbeats to be lost before eviction, not one.

session.timeout.ms is not entirely yours to choose. It must land between the broker's group.min.session.timeout.ms (default 6000) and group.max.session.timeout.ms (default 1800000, 30 minutes), or the JoinGroup is rejected outright. The 45-second default is itself recent: Kafka 3.0 raised it from 10000 via KIP-735, specifically to tolerate transient network blips. If you are on an older broker and seeing eviction storms, check whether you are still running with 10 seconds.

The progress path. max.poll.interval.ms caps the delay between successive poll() calls. Exceed it and the consumer does not wait to be evicted — it proactively sends LeaveGroup. This is the "alive and heartbeating but no longer processing" case, and the client logs it as a poll-timeout expiry (the exact wording varies by client and version).

A worked example

A group of 4 consumers reads a 12-partition topic — 3 partitions each. Defaults throughout: max.poll.records=500, max.poll.interval.ms=300000.

Each poll() returns up to 500 records. Processing is an HTTP call to a downstream service, normally 200 ms, so a full batch takes 500 × 0.2 = 100 seconds. Comfortably inside the 5-minute limit. The group is stable for months.

Then the downstream service degrades and p99 latency climbs to 700 ms. A batch that happens to be mostly p99 calls now takes 500 × 0.7 = 350 seconds. That is past 300000 ms. The consumer sends LeaveGroup mid-batch, its 3 partitions are reassigned, and the offsets it was about to commit are rejected — the generation id has advanced, so that commit comes back REBALANCE_IN_PROGRESS or ILLEGAL_GENERATION, exactly as described above. The new owner therefore reprocesses records the old one already handled. Meanwhile the other three consumers are dragged into the rebalance and stop processing too. Their batches now sit idle for the rebalance duration, and if any of them were already running long, they tip over the same limit. One slow dependency becomes a rebalance loop.

The fix is on the batch side, not the timeout side. Drop max.poll.records to 100: worst case becomes 100 × 0.7 = 70 seconds, with 230 seconds of headroom. You could instead raise max.poll.interval.ms, but that directly extends how long a genuinely stuck consumer holds partitions hostage — the timer is the only thing that will ever notice a deadlocked processing thread. Shrinking the batch keeps the detector sharp. Two related knobs shape how much data arrives per poll: max.partition.fetch.bytes (default 1048576, 1 MiB) and fetch.max.bytes (default 57671680, roughly 55 MiB).

Trigger 3: subscription and metadata changes

The group's job is to cover a set of partitions. Change that set and the group must rebalance, even though no member joined or left.

• Calling subscribe() with different topics replaces the subscription and triggers a rebalance.
• Adding partitions to a subscribed topic produces partitions nobody owns.
• Deleting a subscribed topic removes partitions members currently own.
• subscribe(Pattern) re-evaluates the regex against cluster metadata; a newly created matching topic pulls the group into a rebalance. So does deleting one.

How fast does the client notice? metadata.max.age.ms, default 300000 (5 minutes), is the forced metadata refresh interval. That bounds the delay between someone running kafka-topics --create and your regex-subscribed group rebalancing — which is worth knowing when a rebalance appears with no deploy and no failure to explain it. Much like a service registry that clients re-fetch on a timer, the behaviour looks spontaneous only until you find the refresh that drives it.

Two things that look like triggers but sit slightly outside this taxonomy. First, if the coordinator broker fails or the __consumer_offsets partition leader moves, clients must rediscover their coordinator before they can make progress, and a rebalance may follow even though nothing about your application changed. How much of the group's state survives that move depends on the broker version, so treat it as a trigger to recognise in your logs rather than a mechanism with one fixed shape. Second, KafkaConsumer.assign() — manual partition assignment — does not participate in group management at all. No coordinator, no rebalances, and no automatic failover either. It is a real escape hatch when you have your own assignment scheme, and a bad default when you want fault tolerance.

Reducing the blast radius

You cannot eliminate rebalances; you can make them cheap.

Cooperative rebalancing. The original protocol is eager: every member revokes every partition before rejoining, so the whole group stops while one consumer restarts. KIP-429 (Kafka 2.4.0) added incremental cooperative rebalancing and the CooperativeStickyAssignor: members keep partitions that do not need to move, and only the reassigned ones are revoked, at the cost of an extra rebalance round. Kafka 3.0 changed the default partition.assignment.strategy to [RangeAssignor, CooperativeStickyAssignor] (KIP-726) precisely so you can upgrade with a single rolling bounce that just deletes RangeAssignor from the list.

Static membership. KIP-345 (Kafka 2.3/2.4) lets you set a unique, stable group.instance.id per instance — empty by default. The coordinator then recognises the member across a restart: bounce it and rejoin within session.timeout.ms, and the coordinator does not remove it and does not trigger a rebalance. This is the one good reason to raise session.timeout.ms on purpose: the timeout becomes your restart budget. The identity has to be stable across boots to be worth anything — our own recommendation, not something KIP-345 prescribes, is to derive it from stable per-pod identity such as a StatefulSet ordinal rather than anything randomly generated per boot, or you get a new member every restart and lose the benefit.

KIP-848. The next-generation protocol moves assignment computation from the client-side group leader to the broker-side coordinator and removes the global synchronization barrier, making rebalances fully incremental. Select it with group.protocol=consumer instead of classic. In that mode the session timeout and heartbeat interval are broker-controlled, and partition.assignment.strategy no longer applies because assignment is server-side — so several of the knobs above stop being yours. It became generally available in Apache Kafka 4.0.

Symptom → trigger → fix

Symptom in the logs	Trigger	What to change
Poll timeout expiry, member leaves the group	max.poll.interval.ms exceeded	Lower max.poll.records; move slow work off the poll thread
Member evicted, no poll-timeout message	Missed heartbeats past session.timeout.ms	Investigate GC / network; verify heartbeat.interval.ms ≤ ⅓ of session timeout
Rebalance on every deploy	Join + leave per instance	group.instance.id (static membership) + CooperativeStickyAssignor
Rebalance with no deploy, no failure	Metadata change	Check partition counts and regex-matching topic creation; metadata.max.age.ms bounds the delay
ILLEGAL_GENERATION on commit	Member fell behind the current generation	Fix the underlying eviction; commit before long processing gaps
Burst of rebalances at cold start	Members joining an empty group	group.initial.rebalance.delay.ms on the broker

Elsewhere on this blog

Not about Kafka, but about the same question of what your consumers actually run on:

• Docker vs Kubernetes: What's Actually Different
• AWS Lambda vs EC2: Which Should Run Your Workload?