A database password was rotated cleanly in Vault and synced to a Kubernetes Secret within the hour, yet the service went down thirty minutes later with `password authentication failed`. The password was injected via `env.valueFrom.secretKeyRef`. What actually happened, and what are the real fixes?

Environment variables are read once when the container starts and never updated afterward, so the already-running pods kept the old password in memory, and their pooled database connections — already authenticated — kept working fine. Thirty minutes later, HikariCP's max-lifetime retired the first pooled connection and opened a replacement using the still-old password from the environment: authentication failure. As each remaining connection hit its own max-lifetime, the same failure repeated until the pool had drained to zero. The rotation itself, the sync to the Secret object, and Kubernetes' own machinery were all working correctly — the break was that nothing was responsible for getting the new value into an already-running process. Real fixes: mount the secret as a file with a DataSource that reads it on each new physical connection (no restart needed), a rolling restart triggered by the rotation event, or short-lived dynamic credentials from Vault that the application renews itself — and max-lifetime set well below the rotation interval so every connection gets a chance to reauthenticate before the old password is revoked.