Production readiness

Production readiness is the question you ask before a service takes real traffic: when this breaks, will we know, will we be able to see why, and can we undo it? A service can pass every test and still fail all three.

A checklist is the right tool here — not because engineers are forgetful, but because the items on it are exactly the ones that feel optional right up until the incident that needed them. The aviation checklist exists for the same reason, and Atul Gawande's The Checklist Manifesto makes the case for software as well as surgery.

#The questions, grouped by what they protect

Can it be deployed safely?

• Is there a single command, run the same way every time, that deploys it?
• Does a failed deploy leave the old version serving?
• Is rollback a command, and has anyone run it?
• Are schema changes backward compatible, so the old code still works against the new schema during the switch?

Will we know it is broken?

• Is there a health check, and does it check the thing users need, or only that the process is up?
• Are there alerts on symptoms users feel — errors and latency — rather than only on causes like CPU?
• Does an alert reach a person, and does that person know what to do with it?

Can we see why?

• Are logs structured, with a request ID that follows a request across services?
• Are there metrics for the four golden signals: latency, traffic, errors, saturation?
• Can you get a thread dump and a heap dump from production without restarting it?

Will it survive the obvious failures?

• Does every call to a dependency have a timeout?
• What happens when the database is slow — does the service queue forever, or fail fast?
• Is it running more than one instance, and does losing one matter?
• Is there a backup, and has it been restored?

Is it safe?

• Are secrets in a secret store rather than in the repository or the image?
• Is every input validated, and is authorization checked on the server?
• Are dependencies scanned?

None of those is exotic. The value is in asking all of them, of every service, before launch rather than after.

#Readiness as code, not as a document

A checklist in a wiki is read once, at launch, and is out of date by the next quarter. The stronger form is readiness that runs: checks a deploy cannot skip. This site's deploy script is a small, real example, and each of its guarantees came from a failure.

It checks locally before it touches the server. The whole test suite runs first, and a failure stops the deploy before anything is uploaded.

A failed deploy cleans up after itself. Here is what one looked like on this site, caused by a flaky test:

✗ fix-the-race-condition — THE STARTER PASSES EVERY CASE
  deploy failed before the switch — removing /srv/code10x/releases/20260912-204250

The live site never noticed. Every step up to the build happens in a new release directory while the old release keeps serving, and only a finished build moves the current symlink.

The cleanup itself exists because of an earlier incident, recorded in the script's comments: four failed uploads left four empty release directories, the next successful deploy kept "the five newest" — including all four empty ones — and deleted the only real previous release. A rollback at that moment would have pointed the site at an empty directory.

Rollback checks that it is rolling back to something real. It picks the newest release that is not live and contains a finished build, skipping any that do not. That rule also came from a failure: a failed build's directory had been next in line, and rolling back to it would have reproduced the outage.

#The health check that could not see half the system

The same script contains an honest limitation, which is the most instructive part of it. After the switch it restarts both the web app and a background worker, then checks health — and its comment says plainly that the health check only probes the web app. A worker left running the old release's code against the new database would not show up.

That is the general lesson about health checks:

• A check that the process answers tells you it started. It does not tell you it can reach the database, or that the background jobs are running.
• A deep health check that tests every dependency has the opposite problem: when the database blips, every instance reports unhealthy at once, and a load balancer that trusts it takes the whole fleet out.

The usual answer is two checks with two jobs. A liveness check — "is this process stuck?" — stays shallow, because failing it means restart. A readiness check — "should this instance get traffic?" — can look at dependencies, because failing it means only "route around me for now". Kubernetes names them exactly that, and a restart loop caused by a liveness probe that checks the database is a classic.

#The review, and who says no

Readiness reviews work when someone other than the team that built the service asks the questions, and when "not yet" is an allowed answer. Google's SRE book describes a version in which SRE teams take on a service only after it meets their bar. You do not need an SRE team to borrow the idea: the questions above, asked by one engineer who did not write the service, catch most of it.

The failure mode is the review that becomes a formality — every box ticked, nothing checked. The defence is to demand evidence rather than answers: not "yes, we have rollback" but "show me the last time it was run".