Two copies of the same Kafka record — a genuine redelivery after a rebalance — arrive at a consumer's idempotency check at nearly the same instant, on different threads. Why does a plain `SELECT` check for 'already processed' fail to prevent a duplicate charge, and what actually closes the gap?

Check-then-act — look up whether the payment id was already processed, and if not, charge and record it — has a window between the check and the write, and a redelivered copy that arrives inside that window sees the same "not processed yet" state the first copy saw, because neither has committed its record yet. Both pass the check, both charge. This isn't a rare edge case: it's the direct consequence of two concurrent operations reading a value before either has written it, the same class of bug as an unsynchronised read-modify-write on a shared counter. What closes it is inverting the order — claim the payment id first, atomically, in an operation only one caller can win (a unique constraint), and charge only if the claim succeeded; the loser gets a constraint violation instead of a charge.