Code10x
Configuration and probesmedium3-5 years

A teammate wires the Postgres connection check into `/actuator/health/liveness` so "the pod restarts if the database is unreachable." The database has a rough five minutes one afternoon. What happens to the service, and what should the probe wiring actually look like?

It turns a database incident into a restart storm on top of it. Liveness failing means the kubelet restarts the container — the pod is torn down and started fresh — which does nothing to fix an unreachable database and instead throws away every in-flight request the pod was actually handling, cycles the JVM's warmup, and does this repeatedly for as long as the database stays down, since a fresh container immediately fails the same liveness check again. What should have happened is readiness failing, not liveness: readiness removes the pod from the Service's endpoint list so no new traffic reaches it while the dependency is down, without touching the running process at all, and the pod goes straight back to serving traffic the moment the check passes again — no restart, no lost state, no warmup. The fix is putting the database check behind /actuator/health/readiness and keeping liveness limited to whether the JVM itself is wedged.

The lesson behind it →
PreviousA curious engineer runs `ping` against a Service's ClusterIP from inside the cluster and gets no reply, but `curl`ing the same IP on the Service's port works fine. Is something broken?Next `kubectl describe pod` shows `Last State: Terminated, Reason: OOMKilled, Exit Code: 137`, but the application's own logs have no stack trace, no exception, nothing. Why is there no Java-side evidence, and does the pod's QoS class have anything to do with why it was this pod that got killed?
More on Configuration and probes
A ConfigMap consumed as environment variables is updated to fix a bad setting, but the running pods keep behaving as if nothing changed, even hours later. Separately, a colleague is confused that `kubectl get rs` shows two ReplicaSets with nonzero replicas mid-rollout and assumes the deploy is stuck. Explain both, and design the fix for the first one.