Rate limiting between services

A rate limit is a promise about capacity: this service will do this much, and beyond that it will say no quickly instead of doing everything badly. Between services it is the mechanism that stops one caller, one tenant or one runaway job from consuming what the others need. This lesson is where a limit belongs, the token bucket and Resilience4j's implementation, the distributed version, backpressure as the alternative that asks instead of refusing, fairness between callers, and how to see a limit working before a user tells you.

#Where to limit

A limit at the gateway is about clients and quotas. A limit inside the mesh is about capacity: the inventory service can serve 2,000 requests per second, so it says so, and a caller that sends 5,000 gets 3,000 rejections rather than 5,000 slow responses. The caller-side limit is the polite version of the same thing: a client that knows its dependency's capacity, and its own fair share of it, does not send what will be refused.

#The token bucket

Most limiters are a token bucket: a bucket holds up to capacity tokens, refills at rate tokens per second, and each request takes one. A full bucket allows a burst of capacity requests at once; a steady stream is limited to rate. Two numbers, and they mean different things: rate is the sustained throughput the service can handle, capacity is how big a spike it will absorb. A sliding-window counter (N requests in the last minute) is simpler and allows a burst of 2N across a window boundary; a leaky bucket smooths output to exactly rate with no burst. Token bucket is the default choice because real traffic is bursty and services tolerate short bursts better than they tolerate a smoothing queue.

resilience4j.ratelimiter:
  instances:
    inventory-client:
      limitForPeriod: 100                  # permits per refresh period
      limitRefreshPeriod: 1s
      timeoutDuration: 50ms                # a call waits this long for a permit, then RequestNotPermitted

@RateLimiter(name = "inventory-client", fallbackMethod = "tooBusy")
public Stock stock(String sku) { return rest.get().uri("/stock/{sku}", sku).retrieve().body(Stock.class); }
 
Stock tooBusy(String sku, RequestNotPermitted e) { return Stock.unknown(sku); }

Resilience4j's limiter is local: each instance of the caller has its own bucket, so ten instances with a limit of 100 send up to 1,000. That is fine for a caller protecting a dependency from itself, as long as the number is set per instance with the instance count in mind. It is not a quota.

#Distributed limits

A quota ("this tenant gets 1,000 requests per minute across everything") needs one bucket that every instance shares, and that means a store: Redis, with an atomic script that checks and decrements in one round trip. Bucket4j provides token buckets over Redis, Hazelcast and JDBC; Spring Cloud Gateway's RequestRateLimiter ships with a Redis implementation keyed by whatever the route decides (API key, user id). The cost is a Redis call per request on the hot path, which is a millisecond and a dependency on Redis; when Redis is unavailable, the limiter must decide to fail open (allow) or closed (deny), and that is a decision to make on purpose. Fail open for quotas, fail closed for limits that protect a fragile dependency.

The response to a rejected request is 429 Too Many Requests with a Retry-After header, and the caller's retry logic honours it. Between services the header is what turns a limit into cooperation: the caller backs off for exactly as long as the callee asked.

#Backpressure

A rate limit refuses work. Backpressure slows the producer of the work down before it is produced, so nothing has to be refused. It exists wherever there is a bounded buffer whose fullness reaches back to the source:

• A bounded queue in front of a worker pool. When the queue is full, the submitter blocks or is rejected (CallerRunsPolicy, from the Concurrency course, makes the submitter do the work itself, which slows submission).
• Reactive streams demand. A subscriber requests N items; the publisher sends no more than N until asked again. Project Reactor and the Flow API make this explicit; a slow consumer throttles a fast producer by not requesting.
• TCP itself. A receiver that stops reading fills its window and the sender stops sending. HTTP/2 flow control does the same per stream.
• Kafka, naturally. A slow consumer does not slow the producer, but it does not lose anything either: lag grows, the queue absorbs the burst, and the consumer processes at its own pace. The limit is the topic's retention, and the pressure shows as lag, which is the metric to watch.

Backpressure is preferable wherever the work can wait, because a queued request eventually succeeds and a rejected one does not. It is unusable where the work cannot wait: a user on a page will not wait ten minutes for a queue to drain, so at the user-facing edge the honest tool is a limit and a fast 429, and the queue is reserved for the asynchronous work behind it.

#Fairness

One bucket for all callers means the loudest caller starves the rest: a batch job sending 900 of the 1,000 permits per second leaves the interactive callers fighting over 100. Fairness is buckets per caller (per calling service, per tenant, per API key) with a share each, and a small shared reserve for bursts. It is also priority: when the service is near capacity, the checkout path keeps its permits and the reporting path is the one that gets 429s. Priorities need a signal on the request (a header set by the gateway from the route or the client's tier) and a limiter that reads it; at the simplest, two limiter instances, one for high-priority callers and one for the rest, with the reserve sized so that the important callers never touch the shared bucket.

#Observing limits

A limit nobody can see is a limit discovered by a customer. Resilience4j publishes resilience4j.ratelimiter.available.permissions and .waiting.threads; a Redis limiter is instrumented by counting allows and rejects per key. The dashboard has three lines per limiter: permits used against the limit, rejections, and the p99 wait for a permit. The alerts: sustained rejections on a limiter that protects a dependency (the dependency needs capacity, or the caller needs a queue), and a quota consistently at its ceiling for one tenant (they need a bigger plan, or they have a bug). Log a rejection with the key and the limiter name, once per second per key at most, so a burst of 429s does not become a burst of log lines.

#Under the hood: a bucket in a few integers, and a bucket shared through Redis

A token bucket needs no timer. It stores two numbers, tokens and lastRefill, and on each request computes tokens = min(capacity, tokens + (now − lastRefill) × rate), sets lastRefill = now, and then either decrements and allows or refuses. The refill is lazy, so an idle bucket costs nothing, and the arithmetic is a handful of operations. Resilience4j's AtomicRateLimiter is a variant: it divides time into cycles of limitRefreshPeriod, grants limitForPeriod permits per cycle, and stores the state (cycle number, permits left, nanoseconds to wait) in one AtomicReference updated with compare-and-swap, so it is lock-free under contention; timeoutDuration is how long a caller will sleep for a permit that a future cycle will grant, which is why the limiter can smooth as well as refuse. Bucket4j implements the classic bucket with long nanosecond arithmetic and several bandwidths per bucket (100/s and 5,000/min in one bucket), which is how a limit expresses both a burst and a sustained rate.

A shared bucket moves the same two numbers into Redis and makes the read-compute-write atomic with a Lua script (EVALSHA): the script reads tokens and lastRefill from a hash, does the refill arithmetic with the server's time, writes back, and returns allow-or-deny with the remaining count and the wait time, in one round trip and under Redis's single-threaded execution, so two gateway instances cannot both take the last token. Spring Cloud Gateway's RedisRateLimiter ships exactly this script (request_rate_limiter.lua) and sets a TTL on the keys so idle buckets expire. The cost is one Redis command on the request path, about a millisecond in-cluster, and a fail-open or fail-closed decision that the gateway makes on a Redis error: its default is to allow, which is right for quotas and wrong for a limit protecting a fragile dependency. The sliding window log alternative stores a timestamp per request in a sorted set and counts the last minute with ZCOUNT; exact, but memory proportional to the rate, which is why the sliding window counter (two fixed windows weighted by overlap) is what most gateways use for per-minute quotas.

Backpressure in a bounded executor is ArrayBlockingQueue plus a rejection policy: offer() fails when the queue is full and the policy decides whether the submitter blocks (CallerRunsPolicy runs the task on the submitter's thread, which slows submission by exactly the task's duration), is rejected (AbortPolicy), or the oldest waits are dropped. In Reactive Streams the mechanism is Subscription.request(n): the subscriber says how many it can take, the publisher's onNext calls never exceed the outstanding demand, and an operator between them that cannot honour that (a hot source) must buffer, drop, or error, which Reactor exposes as onBackpressureBuffer, onBackpressureDrop and onBackpressureError. Kafka's version is structural: the consumer pulls with poll(), so the producer is never slowed and the broker's retention is the buffer; the pressure is visible only as lag, which is why lag is the metric.

#Walkthrough: the quota that let every tenant through

A multi-tenant API enforced 1,000 requests per minute per tenant at the gateway, and one tenant was observed doing 6,000 during an incident with no 429s in the logs.

- id: reports
  uri: http://reports:8080
  predicates: [ Path=/api/reports/** ]
  filters:
    - name: RequestRateLimiter
      args: { redis-rate-limiter.replenishRate: 16, redis-rate-limiter.burstCapacity: 1000, key-resolver: "#{@tenantKeyResolver}" }

1. The bucket was right on paper: refill 16/s (about 1,000/min) and a burst of 1,000. The tenant's 6,000 came in a single minute, so the bucket should have refused 5,000.
2. Redis had been failing over during the incident. The gateway's RedisRateLimiter caught the connection errors and, by default, allowed every request (isAllowed returns allowed on error, logging at debug). Six gateway replicas, each failing open, and the reports service received six times its quota from one tenant while its own tenants queued behind.
3. The second finding was the key resolver: it read the tenant from a header the tenant supplied. During the incident the tenant's client had retried with a slightly different header casing, and the resolver was case-sensitive, so half the requests were counted under a second key with its own fresh bucket. Two buckets, one tenant.
4. Fixes: key from the validated JWT's tenant claim, not from a client header; deny-empty-key: true; and a wrapper on the limiter that fails closed for routes marked as protecting a fragile upstream and open for pure quota routes, with a metric on each fail-open event so the next Redis failover is visible as a limiter outage rather than as an upstream overload. Redis itself moved to a replicated setup with the gateway's client set to a short timeout, since a slow limiter is a slow gateway.
5. The postmortem also added a tenant-level dashboard: permits used against the limit per key, rejections per key, and a "fail-open" counter, so the next question a customer asks ("why was I limited at 14:02?") has an answer.

A limiter is a dependency on the store that holds the bucket. What it does when that store is gone is a decision, and the default is not a decision.

#Try it yourself

#Misconceptions

• "A rate limiter needs a background thread to refill." Refill is lazy arithmetic on the timestamps; an idle bucket costs nothing.
• "Redis makes the limit exact across instances." Only with an atomic script; two separate GET/SET commands race. And the limit is only as available as Redis, so decide fail-open or closed.
• "Backpressure and rate limiting are the same thing." A limit refuses; backpressure slows the source. One is for work that cannot wait, the other for work that can.
• "Kafka gives producers backpressure." It does not; the producer is never slowed. The buffer is retention and the signal is lag.
• "A per-instance limit is a limit." It is a limit multiplied by the replica count, which changes without anyone touching the limiter.

#Going deeper

• Bucket4j documentation, "Token bucket algorithm" and the distributed backends; Spring Cloud Gateway, "RequestRateLimiter" and its Lua script in the source.
• Resilience4j, AtomicRateLimiter source and the RateLimiter documentation's state description.
• Reactive Streams specification (the Subscription.request rules) and Project Reactor reference, "Backpressure and ways to reshape requests".
• Cloudflare, "How we built rate limiting capable of scaling to millions of domains", the sliding window counter.
• Stripe, "Scaling your API with rate limiters", on limiter types, fairness and shedding priorities.