On-call and runbooks

On-call is the arrangement that makes sure someone will respond when production breaks at 3am. Done well, it is a quiet week with a phone that rarely rings. Done badly, it is how good engineers burn out and leave — and that is not a figure of speech; it is one of the most common reasons people give for leaving operations-heavy teams.

The difference is almost entirely in things decided before the page arrives.

#Every page must be worth waking up for

The rule that matters most, and the one broken most often: an alert that pages a human must be urgent, real and actionable. If the right response is "look at it in the morning", it is not a page — it is a ticket. If the right response is "nothing, it recovers", it should not exist.

Each unnecessary page costs twice. Once in the sleep it took, and again in what it teaches: a person paged for nothing ten times learns to acknowledge and go back to sleep, and on the eleventh night the page is real.

The observability course's alerting lesson covers how to build alerts on SLOs and burn rates. From the on-call side, the test is simpler. After every page, ask:

1. Was a user affected, or about to be?
2. Did the person paged have to do something?
3. Could they tell what to do?

Three yeses is a good page. Anything else is a change to the alert, made that week.

#A runbook is the answer written down in daylight

A runbook is the page's instructions, written by someone rested who understood the system, for someone tired who may not. It does not have to be long. It has to answer, in order:

• What does this alert mean? In one sentence, in terms of users.
• How bad is it? How to tell a blip from an outage.
• What do I check first? The dashboard, the log query, the command — exact, copyable.
• What are the known causes, and the fix for each?
• When do I escalate, and to whom?

Here is a real one, for a real failure on this site. The deploy printed:

✗ fix-the-race-condition — THE STARTER PASSES EVERY CASE
  deploy failed before the switch — removing /srv/code10x/releases/20260912-204250

A runbook entry for it:

MEANING   A pre-switch step failed. The live site is still serving the
          previous release. No user impact.
SEVERITY  Low. Do not roll back — nothing was switched.
CHECK     Which step failed: the line above "deploy failed before the switch".
          A guard name (✗ ...) means a check failed, not the server.
KNOWN     A concurrency fixture whose starter passed: a race that did not
CAUSES    manifest on that run. Re-run the one guard locally; if it fails
          locally too, the fixture is wrong, not flaky.
          Anything at "Building" — read .build.log in the release directory.
ESCALATE  Only if the site health check itself fails.

Notice the most useful line: "Do not roll back — nothing was switched." The instinct during a failed deploy is to roll back, and here that would change a working site for no reason. A runbook's job is often to stop the wrong action as much as to prompt the right one.

#Runbooks decay; automate the stable parts

A runbook step that says "run this command, and if it prints X, run that one" is a script that has not been written yet. The Google SRE book calls repetitive manual operational work toil, and recommends capping it — its teams aim to spend no more than half their time on operational work, so the rest can go into removing it.

The progression is healthy: an incident teaches you the steps, the runbook records them, and the steps that never needed judgement become automation. What is left in the runbook is the part that does need a human.

This site has an example of the last step. Mirroring the curriculum into the database used to be a manual command after every deploy — a step that was easy to forget and silent when forgotten. It is now part of the deploy script, run before the switchover, and a deploy that cannot run it fails rather than going live half-done.

#Running an incident

When a page turns out to be a real incident, the failure is rarely technical skill. It is coordination: three people investigating the same theory, nobody telling customers, a fix applied that nobody else knew about.

The roles that fix it, from the incident command systems that emergency services use:

On a small team one person may hold two roles. The point is that someone is explicitly the one deciding, and that every change to production is announced before it is made — so that when the graph moves, everyone knows why.

Two habits worth keeping from the first minute:

• Mitigate first, diagnose second. Roll back, fail over, turn the feature off. Users care that it works again, not why it broke. The root cause can be found from logs and dumps tomorrow.
• Write the timeline as you go. Times, what was seen, what was changed. Reconstructing it from memory afterwards produces a tidy, wrong story.

#Sustainable rotations

• Handoffs are written. What is in flight, what is flapping, what was changed. A new on-call engineer inheriting a half-finished incident with no notes is how the same mistake happens twice.
• Being paged has a cost that is paid back — time off after a bad night, or compensation. A rotation that treats it as free is one that loses people.
• Every page is reviewed weekly. Not the incidents — the pages. Noisy alerts are fixed or deleted; that review is what keeps the phone quiet.