Timeouts and retries

Every network call is a bet that the other side will answer. A timeout is the size of the bet; a retry is doubling down. Get the first wrong and one slow dependency ties up every thread you have; get the second wrong and a dependency that was briefly overloaded is buried under your retries and stays down. This lesson is the timeouts a call actually has, the two conditions a retry must meet, backoff with jitter, retry budgets, and the storm that turned a thirty-second blip into a two-hour outage.

#Every call has a timeout

A call without a timeout has one anyway: the operating system's TCP keepalive, which is two hours by default. That is the timeout a service with "no timeout configured" is running with, and it is why a downstream that stops responding without closing connections takes the caller's whole thread pool with it within minutes. There are three distinct timeouts, and a call needs all of them:

@Bean
RestClient inventoryClient(RestClient.Builder builder) {
    var factory = new JdkClientHttpRequestFactory(
        HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(2)).build());
    factory.setReadTimeout(Duration.ofSeconds(3));
    return builder.baseUrl("http://inventory:8080").requestFactory(factory).build();
}
// and the operation that uses it is bounded as a whole:
// Resilience4j TimeLimiter, or the caller's own deadline

The read timeout is set from the dependency's measured latency, not from a feeling. If inventory answers in 40 ms at p99, a 3-second read timeout is generous; a 30-second one means a caller waits 30 seconds to learn what it could have known at 200 ms. And the total must fit inside the caller's own budget: a request that the load balancer abandons at 30 seconds cannot usefully spend 25 of them waiting for one dependency.

Deadline propagation makes the budget follow the request. The gateway allots 5 seconds; the order service spends 1 and forwards a 4-second deadline to inventory; inventory forwards 3 to the warehouse. gRPC carries deadlines natively; over HTTP it is a header the services agree on. Without it, each hop uses its own full timeout and the deepest service is still working on a request whose client hung up a minute ago.

#Retry safely

A retry is only safe when two things are true.

The operation is idempotent. Retrying GET /stock/42 is free. Retrying POST /payments after a timeout may charge the card twice, because a timeout means "I do not know whether it happened", not "it did not happen". Writes become retryable with an idempotency key: the client generates a unique key per operation, sends it in a header, and the server stores the key with the result so a repeat returns the stored result instead of acting again. Stripe's API works this way; so should any endpoint that a caller will retry. Without a key, a write is retried by a human, not by code.

The failure is retriable. A connection refused, a 503, a 429 with Retry-After, a timeout on an idempotent read: worth another attempt. A 400, a 404, a 409, a validation error: the same request will fail the same way, and retrying it is load with no hope.

RetryConfig config = RetryConfig.custom()
    .maxAttempts(3)                                             // the first call plus two retries
    .intervalFunction(IntervalFunction.ofExponentialRandomBackoff(
        Duration.ofMillis(200), 2.0, 0.5))                      // 200 ms, ~400, ~800, each with ±50% jitter
    .retryOnException(e -> e instanceof IOException || e instanceof HttpServerErrorException)
    .failAfterMaxAttempts(true)
    .build();

#Backoff and jitter

Retrying immediately asks a struggling service to do the work it just failed at, right now, again. Exponential backoff spaces the attempts (200 ms, 400, 800) so the dependency has time to recover. Jitter randomises each interval, and it is not optional: without it, every caller that failed at the same instant retries at the same instant, in waves, and the dependency sees its load arrive in synchronized spikes exactly when it is weakest. Full jitter (a random delay between zero and the computed interval) spreads the wave out; the AWS Architecture blog's measurements of this are the standard reference. A Retry-After header from the dependency overrides the schedule: it knows when it will be ready.

#Retry budgets

Retries multiply across hops. If the gateway retries three times, and the order service retries three times, and the inventory service retries three times, one client request that hits a failing warehouse becomes 27 warehouse requests. That is retry amplification, and it is why the rule is retry at one layer only, usually the one closest to the failure, with the layers above it failing fast.

A retry budget caps the damage: retries may be at most, say, 10% of a service's outbound requests over the last minute; past that, further failures are not retried. When a dependency is healthy, the budget is never touched and the occasional transient error gets its retry. When the dependency is down, retries stop being a multiplier and the failure propagates as a failure, which is what it is. Envoy and Google's SRE practice implement budgets natively; in a Spring service it is a token bucket in front of the retry, or simply maxAttempts(2) and a circuit breaker.

#The retry storm

A payment provider's latency rose from 200 ms to 6 seconds for thirty seconds during a failover on their side. The checkout service had a 10-second read timeout and three retries with no backoff. Each checkout request now held a thread for up to 40 seconds; the thread pool of 200 was full in under a minute; every request to the checkout service, including ones that never touched payments, queued and timed out at the load balancer. The clients (a web app and a mobile app) retried their failed requests, three times each, immediately. The provider's failover finished at second 30, and their service came back to find nine times the normal request rate from this one customer, went into overload, and shed load with 503s, which the checkout service retried. The two systems held each other down for two hours, until someone turned off the retries.

Every piece of this was a configuration: a read timeout ten times the p99, retries without backoff or jitter, retries at three layers, and no circuit breaker. The next lesson is the breaker; the rest is this one.

#Hedged requests

For idempotent reads where tail latency matters more than load, a hedged request sends the same request to a second instance if the first has not answered within, say, the p95, and takes whichever returns first. It cuts p99 latency at the cost of a few percent extra load, and it must be capped (hedge at most once, and only below a budget) or it is a retry storm with a different name.

#Under the hood: where each timeout lives, and how retries multiply

The three timeouts are enforced by three different pieces of code. Connect is the socket's connect() with a deadline, in the JDK HttpClient or Apache's connection manager, and it covers DNS resolution only if the resolver itself has a timeout (the JDK's does not by default; a hung DNS server is a hang before the connect timer starts). Read is SO_TIMEOUT semantics: the longest the client will block in a single read() waiting for the next bytes, so a server that trickles one byte every second never trips a 3 s read timeout; the JDK HttpClient's timeout() on the request is better, bounding the wait until response headers arrive, and the body read is bounded only by the total. Total is nobody's by default: Resilience4j's TimeLimiter wraps the call in a CompletableFuture and cancels it after the limit, which cancels the wait but not necessarily the underlying I/O; the JDK client honours cancellation by closing the connection, Apache's does so only with cancel(true) reaching the request. A timeout that fires without cancelling the I/O frees the caller's thread and leaves the connection occupied, which is how a pool empties while every call "timed out".

Deadline propagation is a header (grpc-timeout in gRPC, a custom X-Deadline or Request-Timeout over HTTP) that each hop reads, subtracts its own elapsed time from, and forwards. The deepest service sees a budget that is what is left, not its own default, and a request whose budget has already expired is rejected at the door without work. gRPC's Java client does this for free with withDeadlineAfter; over HTTP it is a ClientHttpRequestInterceptor that reads the inbound deadline from a request-scoped holder and sets the outbound timeout() to the remainder.

A retry budget in Envoy or Finagle is a ratio: retries may be at most 20% of recent requests, tracked over a sliding window per upstream, and a retry that would exceed it is not attempted. Google's SRE version is per-client and per-request: a client-wide budget of 10% and a per-request cap of three attempts, both required. In a Spring service the same thing is a RateLimiter sized to a fraction of the outbound rate, placed before the Retry so it gates only the second and later attempts, plus a circuit breaker that turns a sustained failure into fast rejection with no retries at all. The three compose: the timeout bounds one attempt, the retry bounds attempts per request, the budget bounds attempts per service, and the breaker bounds the time during which any attempts happen.

#Walkthrough: the read timeout that was measured, then wrong

A team sized every read timeout from the dependency's p99 and reviewed them quarterly, as this lesson recommends. One still took them down.

pricing:
  connect-timeout: 1s
  read-timeout: 800ms        # pricing p99 was 120 ms; generous
  retries: 2                 # exponential, jittered, idempotent GET

1. In May, pricing shipped a feature that called a new tax provider for some SKUs; its p99 went to 2.5 s for those, and its p50 stayed at 40 ms, so pricing's own dashboard looked healthy. Orders began timing out 1% of pricing calls, retrying each twice, and failing them.
2. Each timed-out call held an orders worker for 800 ms, then 800 ms, then 800 ms with backoff between: about 3 s per affected request. At 1% of a 500 rps stream, that was five affected requests a second, each holding a thread three seconds: fifteen threads, harmless. Until a flash sale put the tax-provider SKUs at 30% of traffic: 150 requests a second, three seconds each, 450 threads needed, 200 available. Orders fell over; pricing was fine.
3. The TimeLimiter cancelling the wait did not cancel the connection: Apache's client without cancel(true) left the sockets open until pricing answered at 2.5 s, so the connection pool to pricing (50) was exhausted before the thread pool was, and every pricing call, fast or slow, queued for a connection. The symptom moved from "some SKUs slow" to "all of checkout slow".
4. Fixes, in order: the timeout on the JDK HttpClient request (which closes the connection on cancellation), a slowCallDurationThreshold on the pricing breaker so the slow fraction opened it and the fallback (last known price, labelled) took over, and a per-SKU-class split of the pricing call so the tax-provider path had its own bulkhead of ten. The read timeout itself stayed at 800 ms; it had been right, and the world changed.
5. The lasting change was contractual: pricing's API now published a latency SLO per endpoint, orders' timeout was derived from it in code review, and a change to the SLO opened a pull request on the callers.

A timeout sized from last quarter's p99 is a bet on last quarter. The breaker's slow-call threshold and the bulkhead are what make the bet survivable when it loses.

#Try it yourself

#Misconceptions

• "A read timeout bounds the request." It bounds the wait for the next bytes; a trickling response never trips it. Use a per-request timeout for headers and a total for the operation.
• "A timeout frees the resources." It frees the caller's wait. Unless the I/O is cancelled, the connection stays occupied until the server answers.
• "Each layer retrying a little is fine." Retries compound: three layers of three is twenty-seven. One layer, closest to the failure, under a budget.
• "Retry on any 5xx." 501 and 505 will not change; 503 and 502 may. And a timeout on a non-idempotent write is "unknown", not "failed".
• "A measured timeout is a correct timeout." It is correct until the dependency changes. The slow-call breaker and the bulkhead cover the gap.

#Going deeper

• Marc Brooker, "Exponential Backoff And Jitter" and "Timeouts, retries, and backoff with jitter" (AWS Builders' Library).
• Google SRE book, chapter 22, "Addressing Cascading Failures": retry budgets and deadline propagation.
• Envoy documentation, "Retry budgets" in the router filter, and gRPC's "Deadlines" guide.
• Resilience4j documentation: Retry, TimeLimiter, and the "aspect order" section for how the annotations nest.
• Stripe API documentation, "Idempotent requests", the reference design for retryable writes.