Capacity planning and load testing

Capacity planning answers one question before it becomes an incident: how much traffic can this take, and what happens just before it cannot? The second half is the one people skip, and it is the one that matters, because systems do not fail at 100% — they fail well before it.

#The curve nobody expects

Four workers, each request taking 10 milliseconds, so the theoretical capacity is 400 requests a second. Offer load at rising fractions of that and measure latency:

###    utilisation    offered/s     p50 ms     p99 ms
###            50%          200         10         12
###            70%          280         11         12
###            80%          320         11         14
###            90%          360         63        110
###            95%          380        218        397
###            99%          396        236        515

Read it slowly, because it is the most important table in this course.

From 50% to 80% utilisation, latency barely moves. From 80% to 90% — twelve percent more traffic — the p99 goes from 14 ms to 110 ms. At 95% it is nearly 400 ms. The service time never changed. Every request still takes 10 ms of actual work. The rest is waiting in a queue.

That is queueing theory, and the shape is universal: latency stays flat until utilisation approaches capacity, and then rises steeply — a hockey stick, not a slope. The exact knee in that measurement is noisy, because the load generator paces with Thread.sleep and runs in a container; the shape is not noise, and it holds for a thread pool, a database connection pool, a CPU and a checkout queue in a shop.

#Why the averages lie

In that table, the p50 at 90% utilisation is 63 ms and the p99 is 110 ms. An average would sit somewhere in between and look acceptable. Plan on the tail.

• A user loading a page that makes twenty requests hits the p95 of at least one of them most of the time. Tail latency is the latency your users actually experience.
• The requests in the tail are usually the expensive ones — the big customer, the long report — so the tail is disproportionately your most valuable traffic.

#Estimating before you measure

The system design course's estimation lesson gives the arithmetic. For capacity it reduces to four questions:

1. What is the peak, not the average? Traffic is not flat; the daily peak is often three to ten times the mean, and a marketing email or a sale is a multiple of that.
2. What does one request cost — in CPU milliseconds, database queries, connections held?
3. What is the bottleneck? It is almost never the application servers, which scale by adding more. It is the thing that does not: the database's connections, a downstream service's rate limit, a single lock.
4. What is the growth, and how long does it take to add capacity? If provisioning takes a week, you need to know a week before you run out.

Question three decides everything. Doubling your application servers does nothing if they are all waiting on the same database connection pool — the connection pooling lesson's point about pool size being a latency knob, seen from the capacity side.

#Load testing, and the three ways it lies

A load test is how you replace the estimate with a number. It is also easy to run one that produces a comforting and wrong number:

• Coordinated omission. A load generator that waits for each response before sending the next request slows down exactly when the system slows down — so it stops measuring the latency real users would see. Real users do not wait politely. Use a tool that sends at a fixed rate regardless of responses; wrk2, Gatling and k6 in open-model mode do.
• Unrealistic data. Testing against an empty database, or one customer, or a warm cache. The query that is fast on a hundred rows is the one that falls over on ten million.
• Testing in the wrong place. A load test from your laptop measures your laptop's network. Test from inside the same network, against an environment shaped like production.

#What to actually measure

Not "how many requests per second before it errors". Measure:

• The throughput at which your latency SLO breaks. If the promise is p99 under 200 ms, capacity is the load at which p99 crosses 200 ms — which, in the table above, is well before any error.
• What fails first, and how. Does it slow down gracefully, shed load, or fall over and stay down?
• Recovery. When the load drops, does it come back on its own, or does a backlog keep it pinned?

The last one is underrated. A system that recovers by itself after a spike is operationally very different from one that needs a restart — and you only find out which you have by pushing it past its limit on purpose, somewhere it is safe to do so.