Kafka lag and consumer failures
On a real broker: lag growing on every partition, a consumer with zero partitions, a poll interval that failed every commit, and a poison pill stuck at offset 100.
Consumer lag is the number of messages that have been written to a partition and not yet processed by a consumer group. It is the single most useful number for a Kafka-based system, because it measures the thing users eventually feel — how far behind reality the system is — and because its shape usually tells you which of a handful of failures you are looking at.
This lesson runs those failures on a real Kafka broker (a single-node apache/kafka container) with a three-partition topic, and reads each one with kafka-consumer-groups.sh, the tool that ships with Kafka.
Reading lag
For every partition a group consumes, the broker knows two offsets: the log-end offset, where the next message will be written, and the group's committed offset, where it will resume. Lag is the difference. kafka-consumer-groups.sh --describe prints all three.
Three shapes cover most incidents:
| lag shape | usually means |
|---|---|
| rising steadily on every partition | the group is slower than the producers — capacity |
| rising on one partition, zero elsewhere | that partition is stuck — a poison message, or a hot key |
| a sawtooth that resets and climbs again, with duplicate processing | the group keeps rebalancing and redoing work |
And one non-lag symptom: a consumer that is running, healthy, and doing nothing.
The consumer that is simply too slow
A producer writes 300 messages a second. The consumer group invoice-writer spends 10 ms on each message — a database write, say — so it can handle about 100 a second. Two snapshots, fifteen seconds apart:
--- t=10s
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
orders 0 4225 4951 726
orders 1 2037 2621 584
orders 2 3238 4437 1199
--- t=25s
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
orders 0 4824 6396 1572
orders 1 2479 4364 1885
orders 2 3604 6249 2645Total lag went from 2,509 to 6,102 in fifteen seconds. The committed offsets advanced by 1,407 in that time — about 94 messages a second processed, against 300 arriving. Lag grows on every partition, at roughly the difference between the two rates, and it will keep growing until one of them changes.
Two numbers make lag actionable:
- Lag in time, not messages. 6,000 messages is nothing for a topic doing 50,000 a second and hours for one doing two. Divide by the consumption rate, or alert on the age of the oldest unprocessed message, as the backpressure lesson recommends for any queue.
- Is it still growing? A burst that the group is draining is fine. Lag that grows through a quiet period is a capacity problem that will not fix itself.
Adding consumers, and the one that does nothing
The obvious fix for a slow group is more consumers. Here is a group with four consumers on a topic with three partitions:
GROUP CONSUMER-ID HOST CLIENT-ID #PARTITIONS
email-sender email-sender-4-c086117c-c421-43d2-9aab-c5c3b716fdee /127.0.0.1 email-sender-4 0
email-sender email-sender-2-8c59d571-bf33-4197-aa41-e8d2d0ed8194 /127.0.0.1 email-sender-2 1
email-sender email-sender-1-e604420e-65f0-4656-a22e-cff98ec5e843 /127.0.0.1 email-sender-1 1
email-sender email-sender-3-006834b8-9a21-4c34-98d8-4a7344e9fbed /127.0.0.1 email-sender-3 1email-sender-4 has zero partitions. Within a consumer group, each partition is assigned to exactly one consumer, so the partition count is the ceiling on parallelism. The fourth instance is running, connected, passing its health checks, and consuming nothing — while the dashboard counts it as capacity.
When the group is genuinely too slow:
- Make each message cheaper first: batch the database writes, remove a synchronous call, fix the slow query behind the handler.
- Process in parallel within a consumer, with care for ordering — per-key ordering can be kept by routing each key to the same worker.
- Increase the partition count so more consumers can share the work — knowing that it cannot be reduced later, and that it changes which partition a key maps to, so ordering per key is only guaranteed for messages produced after the change.
The rebalance loop
A consumer must call poll() at least every max.poll.interval.ms. If it does not, the group coordinator assumes it has died, removes it from the group, and gives its partitions to someone else. Here the interval is set to 5 seconds, each poll returns up to 500 records, and each record takes 20 ms — ten seconds per batch:
round 1: 500 records in 11.2 s, commit FAILED: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group.
round 2: 500 records in 11.1 s, commit FAILED: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group.
round 3: 500 records in 11.1 s, commit FAILED: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group.
records processed more than once: 500Every batch was processed and no commit ever succeeded. Each time the consumer rejoined, it started again from the last committed offset — the same records — so it did the same work forever, and 500 records were processed more than once. From outside, the consumer looks busy, CPU is high, lag does not move, and every downstream effect of those records happens repeatedly.
The fixes are all about keeping each poll's work inside the interval:
- Lower
max.poll.recordsso a batch fits comfortably inside the interval. - Raise
max.poll.interval.msif long processing is genuinely expected — which also slows the detection of consumers that really have died. - Move slow work off the poll thread, pausing the partitions while it runs, if a batch cannot be made fast.
And handlers must be idempotent, because a rebalance at the wrong moment reprocesses records even when everything is configured well. The idempotency lesson in the distributed systems course covers that.
The poison pill
One message cannot be processed — here, an order whose quantity is the string "two". The handler throws, and the consumer does what many hand-written loops do: seek back to the same offset and try again. The group's partitions afterwards:
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
stock-events 0 100 1000 900
stock-events 1 1000 1000 0
stock-events 2 1000 1000 0partition 0 offset 100: For input string: "two" — retrying
failed attempts on the same record: 30291Partitions 1 and 2 are fully processed. Partition 0 has been stuck at offset 100 since the bad record arrived, with 900 good messages behind it, and the consumer tried the same record 30,291 times in 15 seconds — a hot loop that also hammers whatever the handler calls before it fails.
That is the one-partition shape from the table, and it is diagnostic: find the committed offset on the stuck partition, and read that single record with the console consumer (--partition 0 --offset 100 --max-messages 1) to see what it contains.
The fix is to decide, in code, what happens to a message that fails for a reason retrying will not change:
- Retry a limited number of times with backoff, for failures that may be transient.
- Then send it to a dead-letter topic with the error attached, commit past it, and alert. Spring Kafka's
DefaultErrorHandlerwith aDeadLetterPublishingRecovererdoes exactly this, and it treats deserialization and conversion failures as not retryable by default. - Never retry forever on the same record. A partition stuck behind one message is an outage for every key on that partition.
The retries and dead-letter topics lesson in the Kafka course covers the Spring configuration.
A checklist for a lag alert
- Which partitions? All of them — capacity or a slow dependency. One — a poison pill or a hot key.
- Are committed offsets moving? Slowly — too slow. Not at all with the consumer busy — rebalance loop or retry loop. Not at all with the consumer idle — it is not assigned, or not running.
- Group state and membership.
--describe --membersfor consumers with zero partitions; frequent rebalances in the consumer logs. - What changed? A deploy of the consumer, a slower downstream dependency, a change in the producer's message format.