Failure modesmedium3-5 years

A client calls `POST /transfer` with no idempotency key. The call times out after four seconds with no response. Should the client retry automatically, and what would actually make automatic retry safe?

No — not as the endpoint is built. A timeout is not evidence the request failed; it's evidence the caller stopped waiting, and exactly one of three things actually happened on the server, all indistinguishable from the client's side: the request never arrived (safe to retry, nothing happened); it arrived and completed, but the response was lost on the way back (retrying repeats a real money transfer); or it's still executing past the client's patience (retrying races the original). Nothing local to the client — not the exception, not how long it waited — tells these apart, because the one piece of information that would (did the server finish?) is precisely what failed to arrive. The fix isn't a smarter retry policy on the client; it's making the server endpoint idempotent: the client generates an idempotency key once per logical transfer attempt, sends it on every retry, and the server atomically claims that key before moving money, so a retry either safely does nothing (already completed) or safely proceeds (never started) — whichever of the three actually happened, the retry is correct.

The lesson behind it →