Load balancing and caching
One slow server took round robin's p99 to 556 ms and least-outstanding's to 32 ms; cache layers, and the cache-aside race that caches a stale value after a correct update.
Load balancers and caches are the first two building blocks almost every design reaches for, and for the same reason: they are the cheapest way to handle more traffic than one server can. They are also where a design quietly takes on its first two costs — a routing decision that can concentrate load exactly where it should not, and a copy of the data that can be wrong.
What a load balancer decides
A load balancer sits in front of several identical servers and chooses one for each request. There are two broad kinds, and the difference is how much of the request they read:
| Layer 4 (transport) | Layer 7 (application) | |
|---|---|---|
| sees | IP addresses and ports; forwards connections | the HTTP request: path, headers, cookies |
| can route by | connection | URL, header, user, API version |
| can do | fast, simple forwarding | TLS termination, retries, rate limits, auth at the edge |
| costs | little CPU; blind to requests inside a connection | parses every request; more CPU; more to configure |
A layer 4 balancer that forwards a long-lived HTTP/2 or gRPC connection sends every request on that connection to the same server, however busy it is. That is a common surprise when a service moves to gRPC and its load stops spreading.
The algorithm matters more than it looks
Four servers behind a balancer. Three answer in about 5 ms. The fourth is having a bad day — a garbage-collection storm, a noisy neighbour — and takes about 15 ms. Requests arrive at random, 250 a second. This is a simulation with fixed random seeds, so the same inputs reach every policy and the numbers repeat exactly:
round robin p50 6.0 ms p99 555.8 ms share sent to the slow server 25.0%
random p50 7.8 ms p99 871.3 ms share sent to the slow server 25.0%
least outstanding p50 3.9 ms p99 32.5 ms share sent to the slow server 5.8%
power of two choices p50 5.2 ms p99 68.4 ms share sent to the slow server 17.7%Round robin gave one slow server a quarter of the traffic, and the p99 went past half a second. The slow server can finish about 67 requests a second; round robin sends it 62.5. That is 94% utilisation, which the capacity planning lesson shows is far up the steep side of the latency curve. Its queue, not its speed, is what the p99 measures.
Least outstanding requests — send each request to the server with the fewest in flight — noticed the slow server backing up and sent it under 6% of traffic. The p99 fell by a factor of seventeen, with no change to any server.
Power of two choices — pick two servers at random, send to the less busy — got most of that benefit while looking at only two servers per request. It exists for the realistic case where there are many load balancers, none of which sees the global picture. Checking two random servers is enough to stop piling onto a slow one.
Health checks, and the server that is up but useless
A load balancer removes servers that fail their health checks. The design questions are about what the check tests:
- A check that only confirms the process answers keeps a server in rotation when its database connection pool is exhausted and every real request fails.
- A check that tests every dependency removes every server at once when a shared database has a brief problem, turning a blip into an outage.
The usual answer is the liveness and readiness split described in the production readiness lesson, plus outlier detection: removing a server whose real error rate or latency is far worse than its peers, based on the traffic it is actually serving. That is what would have caught the slow server above even with round robin.
Caching: where the copy lives
A cache keeps a copy of data closer to where it is needed, so the expensive source is asked less often. In a typical service there are several layers, each with a different cost of being wrong:
| layer | holds | typical staleness | invalidation |
|---|---|---|---|
| browser / HTTP | whole responses | as long as Cache-Control says | effectively none — you wait |
| CDN | whole responses, near users | seconds to days | purge by URL or tag |
| application, shared (Redis) | objects and query results | seconds to minutes | delete on write, TTL |
| application, local (in-process) | hot objects | seconds | TTL; each instance separately |
| database buffer cache | pages | always correct | the database's own business |
The further out the cache, the more load it removes and the harder it is to correct. A mistake in a CDN-cached response is served worldwide until a purge completes.
Cache-aside, and the race inside it
The most common pattern is cache-aside: on a read, check the cache; on a miss, read the database and put the result in the cache. On a write, update the database and delete the cache entry so the next read fetches the new value.
It looks correct. It has a race:
reader misses. A request for product 7 finds nothing in the cache, so it goes to the database, which still holds the old price.
reader reads. The reader gets price 100 from the database and is about to put it in the cache — but it is paused here, by a GC pause or a slow network, for a few milliseconds.
writer updates. Meanwhile an admin changes the price. The writer updates the database to 120, exactly as it should.
writer deletes. The writer deletes the cache entry to invalidate it. There is nothing there to delete, so this does nothing at all.
reader writes. The paused reader resumes and puts the value it read — 100 — into the cache. Every request now gets the old price until the TTL expires.
Nothing in that sequence is a bug in any single step. The database was updated. The cache was invalidated. The reader cached what it read. The interleaving produced a cache entry that is older than the database and will stay that way until its TTL expires — and if the TTL is an hour, the wrong price is served for an hour.
The defences, none free:
- Always set a TTL, even with explicit invalidation. It bounds how long any such race can last. The cost is that the TTL also bounds how stale a normal read can be.
- Delete again after a short delay ("delayed double delete"). It narrows the window and does not close it.
- Version the value: write the database row's version into the cache entry, and refuse to overwrite a newer version with an older one. That closes the race, at the cost of a compare-and-set on every cache write.
- Do not cache it. For data where a stale read is a correctness problem — a price at checkout, a balance, a permission — read from the source. The checkout-and-inventory design treats "in stock" on a product page as a hint precisely so the cache is allowed to be wrong there and nowhere else.
The stampede
One more cache failure is worth knowing by name. A popular entry expires, and in the same moment a thousand requests miss and all go to the database for the same value. The database, sized for a cache that absorbs most reads, receives a burst it cannot serve.
The fixes are request coalescing (let one request refresh the value while the others wait for it), refreshing popular entries shortly before they expire, and adding a little randomness to TTLs so that entries written together do not expire together. The Redis and caching course covers each in detail.