Code10x
Delivery guaranteessenior8+ years

A team advertises their Kafka pipeline as 'exactly-once end to end.' What does Kafka's exactly-once actually cover, and where does that claim break down?

Kafka's own exactly-once is narrower than the phrase suggests: it covers a consumer reading from Kafka and a producer writing back to Kafka in the same application, using a transactional producer so the output records and the consumed input's offset commit become visible atomically — a crash never produces duplicate output or silently skips input, for that one Kafka-to-Kafka hop. What it does not cover is anything outside Kafka: a database write, an HTTP call, an email send. Those still need an idempotent consumer — a dedupe key stored in the same transaction as the side effect — because "exactly-once" in Kafka means exactly-once between topics, and end to end it's honestly at-least-once delivery with idempotent effects, not a stronger guarantee than that.

The lesson behind it →
PreviousA lag alert fires. How do you tell from the lag data alone whether it's a capacity problem, a stuck partition, or a rebalance loop?Next Two copies of the same Kafka record — a genuine redelivery after a rebalance — arrive at a consumer's idempotency check at nearly the same instant, on different threads. Why does a plain `SELECT` check for 'already processed' fail to prevent a duplicate charge, and what actually closes the gap?