Kafka lag is climbing and the consumers look idle
An order-events consumer group kept up with production for a year. Since a release two days ago lag climbs all day and only clears overnight, when traffic falls. Nothing has crashed and no consumer has been restarted.
There is nothing to run here. Say what you would look at first and why — the order is the answer, not the diagnosis.
Constraints
- The producer is unchanged, so the work arriving is the same work as last week.
- No consumer has crashed or been restarted by anyone.
Hints
Hint 1
Consumers at 25% CPU are not working hard. Something is stopping them, not slowing them.
Hint 2
Forty-seven rebalances in an hour is not a symptom of lag. Ask what a rebalance costs and what causes one.
Hint 3
A consumer is evicted when it does not call poll() in time. Work out what the new processing time does to the poll deadline.
Stuck? The lesson behind this problem: 📨 Kafka with Spring
What the dashboard says
Consumer lag400 → 2.1M, still climbing
Records consumed/sec8,000 → 900
Producer rate8,000/sec, unchanged
Consumer CPU25%
Broker CPU30%
Rebalances, last hour47 (was 0)
Processing time per record180ms → 1.4s
Partitions / consumers12 / 4, unchanged
Every one of these causes real consumer lag. Which one do these numbers actually support?
Pick one first — the answer stays hidden until you do.