Offsetsmedium0-2 years

A team enabled `enable.auto.commit=true` believing it gives at-least-once delivery. What can it actually lose, and why?

Auto commit doesn't commit "every five seconds" the way it sounds — it commits whatever the previous poll() returned, at the start of the next poll(), once the interval has passed, regardless of whether the consumer actually finished processing that batch. If a batch of 100 records is only half-processed when the interval fires and the next poll commits offset 200, and the process crashes on record 160, records 160–199 are marked done and were never touched — silently lost. That's at-most-once behaviour with a delay, not at-least-once.

The lesson behind it →