Failed deployments

Most production incidents begin with a change, and the most common change is a deployment. That is not an argument for deploying less. It is an argument for knowing the ways a deployment fails, recognising each one quickly, and having already decided when to roll back and when to fix forward.

This lesson uses real failures from this site's own deploy history, taken from the commits that fixed them, and one database experiment on PostgreSQL 16 that reproduces the migration failure that surprises almost everyone the first time.

#The build that said it succeeded

On 26 August 2026, release 20260826-020307 of this site reached production with no build in it. The process started, found nothing to serve, exited, and was restarted by systemd — 26 times, until a rollback.

The build had failed. The deploy script believed it had succeeded. The script's first line set set -euo pipefail, which makes a pipeline fail if any command in it fails — but that setting governs the local shell. The build ran on the server over ssh, in a fresh shell that inherited none of it, as:

npm run build 2>&1 | tail -15

Without pipefail, a pipeline's exit status is the last command's: tail's, which was 0. The commit that fixed it verified the difference on the server itself: false | tail -1 exits 0 there, and 1 with pipefail set.

This failure has a general name: a restart loop. In Kubernetes it is CrashLoopBackOff — the container starts, exits, and is restarted with increasing delays. The recognisable signs are the same everywhere:

• the restart count climbing, with the process never staying up long enough to pass a readiness check
• the useful error in the previous run's logs, not the current one's (kubectl logs --previous, or journalctl for a systemd unit)
• a startup failure: a missing file, a missing environment variable, a port already in use, a database the application cannot reach at boot

The fix that stops it recurring is to check the artifact before switching to it. This site's deploy now treats a build as finished only if .next/prerender-manifest.json exists, and rollback uses the same test before choosing a release to go back to.

#The config that was right somewhere else

A deployment can succeed completely and still be wrong, because it runs with different configuration from the place it was tested.

On 25 August, this site's migration step sourced the environment file with the database URL, and the build step did not. The build reads the database to prerender blog pages — and it does not fail without a database, by design, because CI builds without one. So on the server it succeeded and prerendered an empty blog: /blog said "Nothing published yet", and the sitemap contained no blog URLs, while two published articles sat in Postgres. It recovered by itself within five minutes, when the pages revalidated — a window in which a crawler would have been told the blog had no pages.

The same shape appears in every kind of system: a feature flag default that differs between environments, a secret mounted in staging and missing in production, a timeout set in one profile file and not another. The defences:

• Fail at startup when required configuration is missing, rather than falling back silently. A service that refuses to start is a restart loop you notice in a minute; one that starts with a default serves wrong answers for days.
• One source for configuration, read the same way by every step that needs it.
• A post-deploy check that tests the output, not only that the process is up — the blog index having articles, not the home page returning 200.

#The part of the system the health check does not see

On the same day, a third failure was found in the rollback path. The deploy script restarted both the web service and the background worker, but rollback restarted only the web service. systemd resolves a unit's working directory when it starts, so the worker kept running the release being rolled back from, against the current database.

The health check could not catch it, because it only fetched the home page — which is served by the web service. The fix put the worker restart into one function called by both paths. The general point is the one the production readiness lesson makes: know which parts of the system a health check covers, and do not let "the site returns 200" stand in for the parts it does not.

#The migration that locked the table

Schema migrations are the deployment step most likely to cause an outage that nobody predicted, and the cause is usually not the migration's own speed.

An experiment on a table of two million orders. A reporting session opens a transaction, runs a quick read, and is left idle — someone ran a query in a SQL client and went to lunch. Then the deploy runs a migration that adds a nullable column, which in PostgreSQL is a metadata-only change that takes milliseconds:

ALTER TABLE orders ADD COLUMN gift_note text

While it runs, the API issues an ordinary read with a 3-second statement timeout:

--- D1. ALTER TABLE with no lock_timeout, behind an idle-in-transaction session
API read FAILED after 3.00 s: canceling statement due to statement timeout
migration finished after 3.31 s

A simple read by primary key failed, on a table whose migration takes milliseconds. Here is the chain:

1. The reporting session holds an ACCESS SHARE lock on orders — the weakest lock, taken by every SELECT, held until its transaction ends.
2. ALTER TABLE needs an ACCESS EXCLUSIVE lock, which conflicts with everything, so it waits for the reporting session.
3. The API's SELECT needs ACCESS SHARE, which is compatible with the reporting session's lock — but PostgreSQL queues lock requests, and it will not let the SELECT jump ahead of the waiting ALTER.

So every query on the table queues behind the migration, which queues behind a transaction someone forgot. On a busy service, the connection pool fills within seconds and every endpoint that touches orders fails — an outage caused by a migration that would have taken a millisecond.

The same migration, with one line before it:

--- D2. the same, with SET lock_timeout = '2s' before the ALTER
API read finished after 1.70 s
migration FAILED after 2.00 s: canceling statement due to lock timeout

lock_timeout makes the migration give up if it cannot get its lock in time, releasing the queue behind it. The API read still waited — up to the two seconds the migration was allowed to wait — and then succeeded. The migration failed cleanly and can be retried when the blocker is gone. A shorter timeout, with a retry loop, bounds the damage further.

#Roll back, or fix forward?

When a deploy has made things worse, the default should be roll back first. It restores a known-good state in minutes, and the fix can be written calmly afterwards.

Fixing forward is the right call only when rolling back is worse or impossible:

The middle row is why schema changes deserve so much care. A migration that drops a column, renames one, or changes a type removes the rollback option, because the previous release expects the old schema. Expand-and-contract keeps every intermediate state compatible with both releases, so rollback stays available at every step.

A rollback path that has never been exercised is a hypothesis. This site's own root cause analysis lesson traces how a rollback came one command away from switching to a release with no build in it, and why "rollback exists" is not the same claim as "rollback works".

#Post-deploy checks

The minutes after a deploy are when a failure is cheapest to reverse. A deploy should not be called done until:

• The process is up and stays up — not just started, but past its readiness check and not restarting.
• A check exercises real behaviour — an endpoint that reads the database, a page that depends on the build's output.
• The symptom dashboards are compared with before the deploy — error rate, p99 latency, and saturation for the first few minutes of real traffic.
• Every component was actually updated — web, workers, scheduled jobs, consumers — on the version you think.

A canary — sending a small share of traffic to the new release first — turns those checks into a gate: if the canary's error rate or latency is worse than the old release's, stop before everyone gets it.