Networkinghard5-8 years

A pod behind a Service starts failing its readiness probe while keeping its existing long-lived connections open. Precisely what happens to (1) a request already in flight on an existing connection, and (2) a brand-new connection attempt, and why are those two answers different?

They diverge because kube-proxy's rewrite only ever affects the decision for a new connection, and Linux's connection tracking (conntrack) keeps an already-established connection pinned to whatever pod it was originally routed to, independent of anything happening to the Service afterward. When the pod's readiness fails, it's removed from the Service's EndpointSlice, and kube-proxy's next sync deletes that pod's DNAT rule (or removes it from the IPVS server list) — but that only changes which pod a packet opening a new connection gets rewritten to. A connection that was already open has already been DNAT'd once, and conntrack keeps translating every subsequent packet on that specific connection back to the same pod regardless of whether it's still in the Service's rule set, so an in-flight request finishes (or fails on its own) untouched, while every new connection attempt goes only to the remaining ready pods.

The lesson behind it →
More on Networking