Idempotency and deduplication

Every message system you will use in production delivers at least once. Kafka does, SQS does, RabbitMQ does, and so does a plain HTTP client with a retry. "At least once" means exactly what it says: sometimes twice.

So "exactly once" is not something the infrastructure gives you. It is a property you build into the thing that receives the message, by making the second delivery do nothing. That property is idempotency, and the way you implement it decides whether it actually holds.

#Why duplicates are normal

A consumer takes a message, does the work, and then acknowledges it — commits the offset, deletes it from the queue, returns 200. Any failure between the work and the acknowledgement produces a duplicate:

• The consumer crashes after charging the card and before committing the offset. On restart, the message is delivered again.
• A consumer group rebalances while a message is being processed. Another consumer is given the partition and starts from the last committed offset.
• The acknowledgement is sent and lost on the network, and the broker redelivers after a timeout.
• A producer retries a send whose acknowledgement it never saw, and the broker already had it.

None of those is a bug. They are the price of never losing a message, and every "we never get duplicates" is a sentence about a system that has not yet had a bad day. The Kafka course's lesson on offsets shows the crash case in detail.

#Three handlers, one race

Here is a payment consumer. It receives 2,000 payment messages on 8 threads. One message in twenty is redelivered, and — the realistic part — the copy arrives while the first is still being processed, as it does after a rebalance. Charging takes a millisecond, standing in for a call to the provider.

static void charge(String paymentId) throws InterruptedException {
    Thread.sleep(1);                       // the call to the payment provider
    charges.incrementAndGet();
}

No deduplication does what you expect:

no deduplication                 deliveries 2100   payments 2000   charges 2100

A hundred customers charged twice.

Check, charge, then remember is what most people write first. Look up whether the payment was processed; if not, charge and record it:

if (seen.contains(id)) return;       // SELECT ... WHERE payment_id = ?
charge(id);
seen.add(id);                        // INSERT processed(payment_id)

Three runs:

check, charge, then remember     deliveries 2100   payments 2000   charges 2097
check, charge, then remember     deliveries 2100   payments 2000   charges 2100
check, charge, then remember     deliveries 2100   payments 2000   charges 2100

It caught three duplicates out of three hundred. The check is correct and useless: both copies check at the same moment, both see "not processed", and both charge. The window between the check and the insert is the length of the charge, and the duplicate arrives inside it. This is check-then-act — the compound operation the concurrency course shows is not atomic even on a concurrent map — moved into a database.

Claim first, atomically inverts the order. Record the payment id first, in an operation that only one caller can win, and charge only if you won:

if (!claimed.add(id)) return;        // INSERT ... ; unique violation means someone else has it
charge(id);

claim first, atomically          deliveries 2100   payments 2000   charges 2000

Exactly 2,000, in all three runs. In the demo the atomic claim is ConcurrentHashMap.newKeySet().add, which returns false if the element was already there. In a real service it is an INSERT into a table with a unique constraint on the message or payment id — the database decides who won, and the loser gets a constraint violation instead of a charge.

#The claim that is never released

Claim-first has its own failure, and it is the opposite one. What if the charge fails after the claim succeeded?

A second experiment: 2,000 payments, the provider fails on the first attempt for 1 in 50, and the broker redelivers those once.

claim, then charge                 payments 2000   first attempts failed 40   charges 1960
claim, charge, release on failure  payments 2000   first attempts failed 40   charges 2000

Forty payments were never taken. Each redelivery found the claim already present and skipped — deduplication working perfectly, against a message that was never processed. Releasing the claim when the work fails (deleting the row, or marking it FAILED so a retry may proceed) brings it back to 2,000.

That fix covers an exception. It does not cover a crash between claim and charge, where no code runs to release anything. That is why real claim rows carry a state and a lease:

"Probably died" is doing real work in that table. The worker may be alive and merely slow, which is the leader-election problem in miniature — and the reason fencing tokens exist.

#The side effect you cannot put in your transaction

The deduplication row and the database work can commit together. The payment provider cannot join that transaction. So there is always a window where the charge happened and your record of it did not:

1. Claim the payment id.
2. Call the provider. The provider charges the card, and the response times out.
3. You mark the claim FAILED and a retry runs — and charges again.

Your deduplication cannot fix this, because your system genuinely does not know whether the charge happened. Only the provider knows. The answer is to pass the idempotency key downstream: payment providers accept a key per request and return the original result for a repeated key, which is the same protocol the REST course's idempotency lesson builds on the server side. Idempotency is only a system property if every hop with a side effect honours the same key.

#Natural idempotency, when you can get it

Some operations need no deduplication table, because running them twice is harmless:

• Setting a value: UPDATE orders SET status = 'SHIPPED' WHERE id = ? is idempotent. Incrementing one — SET shipped_count = shipped_count + 1 — is not.
• Conditional updates that encode the expected state: SET status = 'SHIPPED' WHERE id = ? AND status = 'PACKED' does the transition once, and the duplicate affects zero rows.
• Upserts keyed by a business id rather than inserts with a generated one.

Designing messages as "the new state is X" rather than "add one" makes most consumers idempotent without any bookkeeping. It is not always possible — a payment is inherently "move this money" — but it is worth asking for every message type.

#How long to remember

A deduplication table grows forever unless you decide otherwise. Retention must be longer than the longest possible redelivery delay: the broker's retention, plus the time a message can sit in a retry topic, plus how long a consumer can be down before someone replays from an old offset. Keep ids for less than that, and a late duplicate is processed as new.