Rate limiting and HTTP caching
Token bucket and sliding window, 429 with Retry-After, ETags and conditional requests, and what a CDN can cache.
Two mechanisms with the same goal: do less work per request, and less work overall. Caching stops you computing an answer somebody already has. Rate limiting stops one client consuming the capacity of all the others. Both are HTTP features, so a client that speaks HTTP gets them without learning anything about your service.
Conditional requests: the 304 that sends no body
An ETag is an opaque identifier for a version of a response. Send one, and a client can ask whether it is still current.
$ curl -i /catalog
HTTP/1.1 200
ETag: "05e99d767478ea6048d9ae8d185ce5188"
Content-Length: 46The client keeps that tag. Next time, it asks:
$ curl -i /catalog -H 'If-None-Match: "05e99d767478ea6048d9ae8d185ce5188"'
HTTP/1.1 304
ETag: "05e99d767478ea6048d9ae8d185ce5188"304 Not Modified, and no body at all. The client uses what it already has.
Be clear about what this does and does not save. Spring's ShallowEtagHeaderFilter, which produced the tags above, hashes the rendered response — so your controller still ran, the database was still queried, and only the bytes on the wire were saved. That is worth having on a slow connection and worth nothing on your server's load.
To save the work as well, the ETag has to be something you can compute cheaply, before doing the expensive part — a version column, a last_modified timestamp, a hash stored beside the row. Then:
String etag = "\"" + catalog.getVersion() + "\"";
if (etag.equals(request.getHeader("If-None-Match")))
return ResponseEntity.status(304).eTag(etag).build();Now the 304 costs one cheap read instead of a full render. Last-Modified with If-Modified-Since is the same idea at second granularity; ETags are more precise and are the better default.
Cache-Control says who may keep it, and for how long
ETag answers "has it changed?". Cache-Control answers "may you avoid asking at all?".
| Directive | Meaning |
|---|---|
no-store | never write this down, anywhere |
no-cache | you may keep it, but revalidate before using it |
private | only the end client, never a shared proxy or CDN |
public | shared caches may keep it |
max-age=60 | fresh for sixty seconds; use it without asking |
must-revalidate | once stale, do not serve it without checking |
The one that matters most in a backend is private. A per-user response cached by a shared proxy is one user seeing another's data, and the default in some setups is more permissive than you expect. Anything personalised is private, and anything with a token in it is no-store.
Rate limiting, and the headers that make it usable
A limiter's job is to protect capacity, and the difference between a good one and a bad one is almost entirely in what it tells the client.
HTTP/1.1 200 RateLimit-Remaining: 2
HTTP/1.1 200 RateLimit-Remaining: 1
HTTP/1.1 200 RateLimit-Remaining: 0
HTTP/1.1 429 Retry-After: 37 RateLimit-Remaining: 0A client can see its budget shrinking and slow down before being refused. When it is refused, Retry-After: 37 tells it exactly how long to wait — without that header a client guesses, and the guess is usually "immediately", which is how a rate limit turns into a retry storm.
Return a Problem Details body with it, as the error lesson established, so the refusal has a type the client can branch on rather than a bare status.
Four algorithms, and the one to reach for
| Algorithm | How it behaves |
|---|---|
| Fixed window | count per clock minute. Simple; allows a double burst across the boundary — 3 at 11:59:59 and 3 at 12:00:00 |
| Sliding window | count over the last 60 seconds, whenever "now" is. Smooth, more state |
| Token bucket | tokens refill at a steady rate; a request spends one. Allows a burst up to the bucket size, then enforces the average |
| Leaky bucket | requests drain at a fixed rate; no bursts at all |
Token bucket is usually right for an API, because real clients are bursty in a legitimate way — a page loads and fires six requests — and what you actually want to prevent is a sustained rate, not a brief clump.
The decision that matters more than the algorithm is what you key on. Per API key or user id is the useful unit. Per IP address punishes an office behind one NAT and does nothing to an attacker with a thousand addresses. Most production limiters key on the authenticated principal and fall back to IP only for unauthenticated endpoints.
Where they meet
The two mechanisms compose, and the combination is what makes a public API survivable:
- A cacheable,
public,max-age'd response never reaches your service at all, so it consumes no rate-limit budget and no capacity. - A conditional request that returns 304 is cheap for you and should usually cost the client less budget than a full response — some APIs do not count them at all.
- A rate-limited client that respects
Retry-Afterrecovers; one that does not becomes the load you were protecting against.
And the ordering matters in the filter chain: rate limiting goes before authentication is expensive but after you can identify the caller — which in practice means after a cheap API-key lookup and before anything that touches a database.