Root cause analysis

An incident is expensive. The root cause analysis is how you get something back for the money: an understanding of why it happened that is accurate enough to change something, and a list of changes that actually get made.

Most write-ups fail one of two ways. They stop at the first cause that sounds sufficient — usually a person's mistake — or they produce a long list of action items that nobody owns and nobody finishes. This lesson is about avoiding both, and it works through a real chain of failures in this site's own deploy script, reconstructed from the commits that fixed them.

#Blameless, and why that is not softness

A blameless postmortem does not ask who made the mistake. It asks why the mistake was possible, and why nothing caught it.

That is not kindness for its own sake. It is the only way to get accurate information. An engineer who expects to be blamed describes what happened in the least damaging way, leaves out the shortcut they took, and does not mention the alert they ignored because it fires every night. Those omissions are exactly the details the analysis needs.

And "a person made an error" is almost never a useful conclusion. People will always make errors. The question is why this one reached production — which review did not catch it, which test did not exist, which safeguard failed silently. Those are things you can change.

#A timeline first

Every analysis starts from a timeline: what happened, when, and what people knew at each point. Written from logs, commit history, chat messages and dashboards — not memory, which reorders events into a story that makes sense.

Here is the one this lesson analyses. It is a near miss: no learner saw an outage. That makes it a better example than most, because the interesting part is how close it came and how many layers were involved.

#Five whys, and where they stop

The classic technique is to ask "why?" repeatedly until you reach something fundamental:

1. Why could a rollback have broken the site? Because it would have switched to a directory with no build.
2. Why was there a directory with no build next in line? Because a failed deploy left it behind.
3. Why did the failed deploy leave it? Because the cleanup trap did not run.
4. Why did the trap not run? Because errtrace was not set, so functions did not inherit it.
5. Why did nobody notice? Because the failure path had never been exercised after the trap was written.

That chain is genuinely useful, and the fifth answer is the most valuable line in the analysis. But notice what a single chain hides. It follows one path, and this incident had several independent causes, any of which removed would have prevented the near miss:

#Contributing factors, not a root cause

Written as contributing factors instead:

• An unreliable network path made deploys fail at the upload step. Outside the system's control — so the system must tolerate it.
• A flaky test failed a build that should have passed. A test that fails randomly is a test people learn to re-run.
• Pruning counted directory names, not usable releases. A safety mechanism measuring the wrong thing.
• Rollback chose by name, not by content. The emergency path trusted state it never checked.
• The cleanup had never run, because of a missing shell option. A safeguard written, believed in, and untested.

"Root cause" in the singular pushes toward picking one of those and stopping. The upload failure would be the natural choice — it happened first — and fixing it alone would have left every other hazard armed for the next unrelated failure, which arrived five days later as a flaky test.

#Action items that get done

An action item is worth writing only if it is specific, owned, and verifiable. "Improve deploy reliability" is none of those. Each factor above became a change whose effect could be checked:

Every one of those is recorded next to the code it changed, in a comment that names the date and the incident. That is the last property of a good action item: the reason survives. The next person who wonders why rollback checks for a manifest file does not have to rediscover the incident, and does not "simplify" the check away.

#The lesson inside the lesson

The most important sentence in this analysis is not about Bash. It is this: the cleanup was written, and a missing option disabled it. The fix on 11 September at 14:54 was correct — it made rollback check what it switches to — and it was one layer above the real defect, which was found three hours later only because someone asked why the trap had not handled it.

Two habits follow:

• Test the failure path, not only the success path. A safeguard that has never fired is a hypothesis.
• When a fix works, ask what should have prevented the problem in the first place, and whether that is still broken.

#The document itself

A postmortem is read by people who were not there. A structure that works:

1. Summary — three sentences: what happened, the impact, the state now.
2. Impact — who was affected, for how long, how badly. "None, a near miss" is a valid answer and worth writing up.
3. Timeline — as above, from records.
4. Contributing factors — the weaknesses, not the people.
5. What went well — the timestamped release design kept the live site untouched through every failure here. Knowing what worked stops it being removed.
6. Action items — specific, owned, verifiable, with dates.

Then share it widely. A near miss written up in one team is a real incident avoided in another.