Senior

Production Engineering

The incidents, and the method that resolves them.

A method for diagnosing a live system, and its application to the classics: latency spikes, connection pool exhaustion, memory leaks, consumer lag, deadlocks, high CPU, OOM, thread pool exhaustion and failed deployments. Each one is shown on a real JVM, database or broker.

13 lessons written3 modules~3h reading

After this course you can

Run an incident: triage, hypothesis, evidence, fix, verification, write-up
Diagnose each of the nine classic backend incidents from its signals
Write a blameless RCA that changes something
Run a service: readiness, capacity headroom, runbooks, restorable backups and an audit trail

Curriculum

13 lessons · outlined lessons show their plan

Method

2/2 written · ~25 min

Diagnosing a live systemWhat, for whom, since when; hypotheses with the evidence that would rule them out; USE for every pool — and a pegged CPU traced to one line with top -H and a thread dump.13 min Root cause analysisBlameless for accuracy, a timeline from records, contributing factors instead of one cause — worked through this site's own near miss, where a cleanup trap had never run.12 min

The classics

5/5 written · ~66 min

Slow APIs and database bottlenecksOn real Postgres: a 27 ms query that became a 218 ms p50 under concurrency, a lock wait named by pg_blocking_pids, and a drained pool that failed every fast query.14 min Memory leaks and OOMA healthy sawtooth against an old-gen floor that only rises, the class histogram that points at the map, the heap dump — and leak versus undersized heap versus OOMKilled.13 min Kafka lag and consumer failuresOn a real broker: lag growing on every partition, a consumer with zero partitions, a poll interval that failed every commit, and a poison pill stuck at offset 100.14 min Thread problemsThree dumps from a live JVM: the deadlock the JVM names, four of four threads parked on a call with no timeout, and five threads blocked behind one slow lock holder.12 min Failed deploymentsThis site's own: a failed build deployed 26 restarts deep, an empty blog from missing config, a stale worker — and a millisecond ALTER TABLE that timed out a SELECT.13 min

Operating a service

6/6 written · ~70 min

Production readinessWill we know it broke, can we see why, can we undo it — the checklist, readiness that runs inside the deploy, and the health check that could not see half the system.11 min Capacity planning and load testingThe latency knee measured — p99 from 14 ms to 110 ms for twelve percent more traffic — headroom, the real bottleneck, and the three ways a load test lies.11 min On-call and runbooksPages worth waking up for, a real runbook for a real failed deploy, toil turned into automation, incident roles, and rotations people survive.12 min Backups and disaster recoveryRPO and RTO first, a backup designed to fail loudly, the same-host gap, 3-2-1, restoring into a new database, and the drill that proves it.12 min Audit loggingWho did what and when, the event lost after commit, append-only, and a hash chain shown catching an edit — and missing a full rewrite without an anchor.12 min Chaos engineeringBreaking your own system on purpose: one hanging call in ten moved the median from 12 ms to 2.5 s, a timeout fixed it, and the steady state, blast radius and abort switch that keep an experiment from becoming an incident.12 min