Backpressure

Backpressure is what happens when a slow part of a system is allowed to tell a fast part to slow down. Without it, the fast part keeps going, and the difference between the two rates has to go somewhere. It goes into a queue. The queue grows, and it keeps growing until something breaks — usually the heap, usually at the worst moment, and usually a long way from the code that caused it.

The idea fits in one sentence and is ignored often enough to cause a large share of real outages: every queue must have a limit, and you must decide what happens when it is reached.

#A queue that grows forever

A producer offers 1,000 items a second. A single consumer takes about 2 ms per item, which in practice worked out at roughly 390 a second. Between them is a LinkedBlockingQueue created with no capacity — which means a capacity of Integer.MAX_VALUE, which means no limit.

Sampled every two seconds:

UNBOUNDED
  t=2s  queued  1223   accepted  2004   rejected     0   latest item waited  1222 ms
  t=4s  queued  2459   accepted  4030   rejected     0   latest item waited  2459 ms
  t=6s  queued  3679   accepted  6032   rejected     0   latest item waited  3678 ms
  t=8s  queued  4863   accepted  7996   rejected     0   latest item waited  4906 ms

Two things grow in a straight line, and both keep growing for as long as the load lasts:

• The queue, by about 600 items every second.
• The wait. An item taken at eight seconds had waited nearly five seconds before anyone looked at it.

Nothing errored. Every item was "accepted". The producer believes everything is fine. If those items are HTTP requests, every one of them is eventually answered — after the client gave up and retried, adding another item to the queue.

A second run reproduced the same numbers to within a few percent, so the slope is the system, not noise.

#And then the heap

Keep going and the queue turns out to be limited after all — by memory. The same unbounded queue, holding 4 KB request bodies, on a 64 MB heap:

t=2s queued 1735
t=4s queued 3447
t=6s queued 5156
...
t=16s queued 13670
t=18s queued 15359
t=18.5s queued 15758  java.lang.OutOfMemoryError: Java heap space

Fifteen thousand queued 4 KB bodies is about 63 MB. The heap was the limit all along; nobody chose it, and reaching it does not reject one request — it takes the whole process down, including every request that was being served perfectly well. The first version of that demo could not even print its own error, because printing needs memory too.

#Bound it, and choose what happens when it is full

The same producer and consumer, with an ArrayBlockingQueue of 200. There are two honest choices for the moment it fills.

Block the producer — put() waits for space:

BOUNDED_BLOCKING
  t=2s  queued   200   accepted   971   rejected     0   latest item waited   518 ms
  t=4s  queued   200   accepted  1736   rejected     0   latest item waited   534 ms
  t=6s  queued   200   accepted  2461   rejected     0   latest item waited   529 ms
  t=8s  queued   194   accepted  3221   rejected     0   latest item waited   517 ms

The queue stops at 200, and the wait stops at about half a second — 200 items at the consumer's pace. The producer now runs at the consumer's speed, about 400 a second, because it is made to. That is backpressure in its purest form: the slow stage's limit has travelled back upstream.

Reject — offer() returns false when full:

BOUNDED_REJECTING
  t=2s  queued   200   accepted   975   rejected  1026   latest item waited   516 ms
  t=4s  queued   200   accepted  1753   rejected  2250   latest item waited   515 ms
  t=6s  queued   200   accepted  2522   rejected  3482   latest item waited   512 ms
  t=8s  queued   198   accepted  3290   rejected  4710   latest item waited   514 ms

The same bounded wait, and the same accepted rate — the consumer's. The excess, well over half the offered load, is refused immediately. That is load shedding: saying no quickly to the work you cannot do, so the work you accept is done in time.

Notice what did not change between the two bounded runs: accepted throughput is the consumer's rate either way. No queue policy makes a slow consumer faster. The only choices are where the excess waits, and whether anyone is told.

#Which one, where

A rejection is not a failure of the design. For a request that would otherwise wait five seconds and then time out, a fast 503 with Retry-After is the better answer: the client learns immediately and can back off, and the capacity that would have gone to work nobody is waiting for goes to requests that can still succeed.

#Backpressure has to reach the start

A bounded queue in the middle of a pipeline only pushes the problem one stage upstream. The stage before it blocks, rejects, or has its own queue, and so on back to the source. The design works only when the limit reaches something that can actually slow down or say no:

• Pull-based consumers get it for free. A Kafka consumer that polls only when it has capacity never takes more than it can process; the lag sits on the broker, on disk, where it is visible and bounded by retention rather than by your heap.
• Reactive streams — Project Reactor, RxJava, java.util.concurrent.Flow — make it explicit: a subscriber requests n items, and a publisher may not send more than was requested.
• HTTP has no backpressure of its own. A server's only ways to push back are limiting concurrent requests and rejecting with 429 or 503, which is why a bounded executor and a rejection path are part of a service's design, not tuning.

#The retry that makes it worse

Retries are backpressure's enemy. When a system is overloaded, requests time out; clients retry; the retries are new requests; the load rises further. A system at 110% of capacity with every client retrying three times is not at 110% for long.

The defences belong together:

• Exponential backoff with jitter on every retry, so retries spread out instead of arriving in synchronised waves.
• A retry budget — retries may be at most, say, 10% of requests — so a failing dependency cannot triple its own load.
• Circuit breakers that stop calling a dependency that is failing, giving it room to recover.
• Deadlines passed downstream, so a service does not start work whose caller has already given up.

Without them, a short overload can become an outage that persists after its original cause has gone. That self-sustaining state has a name — a metastable failure — and the failure modes lesson in this course looks at it directly.