Chaos engineering

Chaos engineering is breaking your own system on purpose, in a controlled way, to find out how it really fails before an outage finds out for you. The name makes it sound reckless. Done properly it is the opposite: it is an experiment with a hypothesis, a small blast radius and an off switch.

Netflix made the idea famous with Chaos Monkey, a tool that terminated production instances at random during working hours, so that every team had to build services that survived losing one. The thinking was later written up as the Principles of Chaos Engineering, and the discipline since then has been less about random destruction and more about careful experiments.

#Why you cannot just reason about it

Everyone believes their service handles a slow dependency. Here is a small one that does not, and the result is not what most people would predict.

A service with four request threads receives 50 requests a second. Each request calls a dependency that normally takes 10 ms. The experiment injects one fault: one call in ten hangs for two seconds.

static String callDependency(int i, boolean fault) throws InterruptedException {
    Thread.sleep(fault && i % 10 == 0 ? 2000 : 10);
    return "ok";
}

Two hundred requests, three runs each, with and without the fault:

baseline, no fault         p50    12 ms   p99    15 ms   degraded   0
fault, no timeout          p50  2528 ms   p99  6908 ms   degraded   0

Read the p50. The fault touched twenty requests — one in ten. The median request went from 12 ms to two and a half seconds, and the slowest took nearly seven.

A tenth of the calls were slow, and every request suffered, because the twenty slow calls each held one of only four threads for two seconds. While they waited, the fast requests queued behind them. A partial failure in a dependency became a total failure of the service, and nothing in the code looks wrong. That is the kind of thing an experiment finds and a design review does not.

The three runs agreed to within a few percent (p50 2522–2561 ms, p99 6858–6935 ms), so the shape is real; the exact numbers belong to one container on one laptop.

#The fix, proved by the same experiment

The service needed a timeout on the call, and something sensible to return when it fires:

Future<String> f = calls.submit(() -> callDependency(id, fault));
try {
    f.get(100, TimeUnit.MILLISECONDS);
} catch (TimeoutException e) { f.cancel(true); degraded[id] = true; }

Same fault, run again:

baseline, no fault         p50    12 ms   p99    15 ms   degraded   0
fault, no timeout          p50  2528 ms   p99  6908 ms   degraded   0
fault, 100 ms timeout      p50    12 ms   p99   105 ms   degraded  20

The p50 is back to 12 ms. The p99 is 105 ms — the timeout itself. Twenty responses were degraded, exactly the twenty calls that hung, and the other hundred and eighty did not notice. The damage is now the size of the fault.

That is the entire loop of chaos engineering in one table: a hypothesis ("a slow dependency will not hurt us"), a fault, a broken hypothesis, a fix, and the same experiment run again to prove the fix rather than assume it.

#The shape of an experiment

Each step exists because skipping it turns an experiment into an incident:

• Steady state is a number users would notice — latency, error rate, orders per minute — not CPU. If you measure the wrong thing, a fault that hurts users looks fine.
• The hypothesis is written first. "We expect the p99 to stay under 200 ms." Without it, any result can be explained after the fact.
• The blast radius starts small. One instance, one percent of traffic, one availability zone, a staging environment first. Widen it only after the small version held.
• The abort condition is agreed before you start, and somebody is watching it. "If errors exceed 1% or p99 passes one second, stop." When it trips, you stop — you do not stay to see what happens next.
• Removing the fault must be faster than adding it. An experiment you cannot switch off is an outage you scheduled.

#What to break

Start with the failures that happen anyway, in roughly this order:

Slow is more instructive than dead. A dependency that is down fails fast, and most code handles fast failure. A dependency that is slow holds resources, and that is the failure most services have never seen — as the experiment above shows.

The tools range from a proxy that injects latency between a service and its dependency (Toxiproxy is the common one) to platform tools that kill pods or partition networks (Chaos Mesh and LitmusChaos on Kubernetes, AWS Fault Injection Service on AWS). You do not need any of them to begin. A test that wraps a client and adds a sleep, as above, answers the first and most important question.

#Doing it responsibly

• Not before the basics. If the service has no timeouts, no alerts and no rollback, you do not need an experiment to know it will fail. Fix the readiness list first; chaos engineering finds the failures that remain after that.
• Tell people. An experiment that surprises the on-call engineer wastes their time and yours. Schedule it, announce it, and run it during working hours with the owners present.
• Game days before automation. A game day is a planned session where a team injects a fault together and watches. It exercises the runbooks and the people as well as the code. Automated, continuous experiments come later, once the manual ones stop finding things.
• Staging first — but staging is not the goal. Staging has different traffic, different data and different configuration, so a fault that is harmless there can still hurt production. Mature teams do run carefully scoped experiments in production, and only because they have the observability and the abort switch to do it safely.