Circuit breakers and bulkheads

Timeouts bound how long one call waits. They do not stop the next thousand calls from waiting the same way. When a dependency is down, the fastest, kindest thing to do is to stop calling it: fail immediately, give the caller something honest, and check back in a while. That is the circuit breaker. And when several dependencies share one thread pool, the slow one takes the pool with it; bulkheads keep each in its own compartment. This lesson is the breaker's three states, its Resilience4j configuration, fallbacks that tell the truth, bulkheads, and shedding load before it sinks you.

#The three states

A breaker wraps calls to one dependency and watches their outcomes over a sliding window.

• Closed. Calls pass through. Failures and slow calls are counted.
• Open. The failure rate (or the slow-call rate) crossed the threshold. Calls fail immediately with CallNotPermittedException, without touching the dependency, for a wait duration. The dependency gets a rest; the caller gets a fast, definite answer instead of a slow, uncertain one.
• Half-open. After the wait, a small number of trial calls are allowed through. If they succeed, the breaker closes; if they fail, it opens again for another wait.

The breaker does not make the dependency work. It makes the caller stop pretending it might, which frees the caller's threads, stops the caller from adding load to a struggling service, and turns a hang into an error the code can handle.

#Resilience4j

resilience4j.circuitbreaker:
  instances:
    inventory:
      slidingWindowType: COUNT_BASED
      slidingWindowSize: 50                    # judge on the last 50 calls
      minimumNumberOfCalls: 20                 # but not before 20 have been made
      failureRateThreshold: 50                 # open at 50% failures...
      slowCallDurationThreshold: 2s            # ...or when calls slower than 2 s...
      slowCallRateThreshold: 80                # ...are 80% of the window
      waitDurationInOpenState: 10s
      permittedNumberOfCallsInHalfOpenState: 5
      recordExceptions: [ java.io.IOException, org.springframework.web.client.HttpServerErrorException ]
      ignoreExceptions: [ org.springframework.web.client.HttpClientErrorException ]

@Service
public class InventoryClient {
    @CircuitBreaker(name = "inventory", fallbackMethod = "stockUnknown")
    public Stock stock(String sku) {
        return rest.get().uri("/stock/{sku}", sku).retrieve().body(Stock.class);
    }
 
    // Same signature plus the exception. Called when the breaker is open OR the call failed.
    Stock stockUnknown(String sku, Throwable cause) {
        return Stock.unknown(sku);              // an honest value the caller can render as "checking…"
    }
}

Three things in that configuration are decisions, not defaults. slowCallDurationThreshold makes the breaker open on latency as well as on errors, which matters because the dependency that hurts you is usually slow rather than dead. ignoreExceptions keeps a 404 or a 400 from counting as a failure: a client error is the caller's fault and says nothing about the dependency's health. And minimumNumberOfCalls stops a breaker from opening on the first two failures after a deploy. One breaker per dependency, named for it, so that inventory being down does not open the breaker for payments.

The Spring Boot starter (resilience4j-spring-boot3) wires the annotation, exposes /actuator/circuitbreakers and /actuator/circuitbreakerevents, and publishes resilience4j.circuitbreaker.state and .calls to Micrometer. When several Resilience4j annotations sit on one method, the order from outside in is Retry, CircuitBreaker, RateLimiter, TimeLimiter, Bulkhead: the retry sees the breaker's rejection and does not retry it, the breaker counts the timeout as a failure.

#Fallbacks that are honest

A fallback is what the caller gets when the dependency is unavailable, and it is where resilience turns into lying if you are careless. The honest options:

• Stale data, labelled. Serve the last known price from a cache and say it is as of 14:02. Right for reads where old is better than nothing.
• A degraded feature. Recommendations are unavailable, so the page renders without them. The rest of the page works.
• A safe default that says so. Stock.unknown() renders as "checking availability", not as "in stock".
• A fast, clear failure. 503 Service Unavailable with Retry-After, in a hundred milliseconds instead of thirty seconds. For a write that cannot be done without the dependency, this is the right fallback, and it is a fallback.

The dishonest one is the fabricated value: returning "in stock" because the inventory service is down, or "payment succeeded" from a fallback, or an empty list that the UI renders as "you have no orders". A fallback that the user cannot distinguish from a real answer is a bug scheduled for later.

#Bulkheads

A ship's hull is divided so that one breach floods one compartment. In a service the hull is the thread pool. Tomcat's 200 threads serve every endpoint; if the endpoints that call the slow reporting service hold 200 threads waiting, the endpoint that only reads from the local database, which needs nothing external, cannot get a thread either. One slow dependency has sunk everything.

A bulkhead caps the concurrency any one dependency may consume:

resilience4j.bulkhead:
  instances:
    reporting:
      maxConcurrentCalls: 10          # at most 10 threads inside the reporting client at once
      maxWaitDuration: 100ms          # the 11th waits this long, then fails with BulkheadFullException

The semaphore bulkhead above limits concurrent entries into the wrapped code on the caller's own thread. The thread-pool bulkhead runs the call on a dedicated pool with a bounded queue, isolating it completely, at the cost of a thread hop and a CompletableFuture return type. With the reporting client capped at ten, a reporting outage costs ten threads and 190 keep serving. Size each bulkhead from the dependency's normal concurrency (rate × latency, Little's law again) with headroom, and treat BulkheadFullException the way you treat an open breaker: fallback or fast failure.

The same principle applies one level down: a separate connection pool for the analytics database so a slow report cannot exhaust the pool that checkout uses, and a separate Kafka consumer group per concern. Anything shared is a compartment that floods together.

#Load shedding

Breakers and bulkheads protect a service from its dependencies. Load shedding protects it from its callers. When more requests arrive than the service can finish, accepting them all means every request gets slower until every request times out and none succeeds. Shedding means rejecting the excess early, with a 503 in a millisecond, so the requests that are accepted finish.

The signals: queue length (Tomcat's accept backlog, the executor's queue), in-flight request count against a limit, or latency against a target. The simplest implementation is a concurrency limit at the edge (a semaphore around the request, sized from the service's measured capacity) that rejects when full; adaptive limiters (Netflix's concurrency-limits library, or the mesh's) adjust the limit from observed latency. Shed the cheap-to-reject work first (unauthenticated, retries, low-priority clients) and keep the expensive-to-lose work (a payment in progress). A service that sheds load stays up at its capacity; one that does not falls over at its capacity plus one.

#Under the hood: the state machine, the window, and the semaphore

A Resilience4j CircuitBreaker is a state machine guarded by an atomic reference, plus a sliding window of outcomes. Every call goes tryAcquirePermission() (which in OPEN throws CallNotPermittedException immediately, and in HALF_OPEN counts against permittedNumberOfCallsInHalfOpenState with an atomic counter), then the call, then onSuccess(duration) or onError(duration, throwable), which records the outcome into the window and evaluates the thresholds. The count-based window is a ring buffer of the last N outcomes with running totals of failures and slow calls; the time-based window is a ring of per-second buckets over the last N seconds, aggregated on each record. The transition CLOSED → OPEN happens inside onError/onSuccess when the window has at least minimumNumberOfCalls and either rate crosses its threshold; OPEN → HALF_OPEN happens on the first tryAcquirePermission after waitDurationInOpenState has elapsed (or on a scheduled transition if automaticTransitionFromOpenToHalfOpenEnabled), which means an open breaker with no traffic stays open, untested, until someone calls. In HALF_OPEN a fresh, small window judges the permitted calls: all recorded, and the rate over just those decides CLOSED or back to OPEN. A call that was in flight when the breaker opened still completes and is recorded; the breaker does not cancel anything.

The semaphore bulkhead is a Semaphore(maxConcurrentCalls) with tryAcquire(maxWaitDuration); the call runs on the caller's thread and the permit is released in a finally. It bounds concurrency, not throughput, and it isolates nothing about where the call runs: a call that blocks holds the caller's thread as before, only fewer of them. The thread-pool bulkhead submits the call to its own ThreadPoolExecutor with a bounded queue and returns a CompletableFuture; the caller's thread is free immediately, the pool's threads are the only ones a slow dependency can hold, and BulkheadFullException is thrown when the queue is full. The trade is the hop, a copy of thread-local context (tracing, security) that must be propagated by hand, and a CompletableFuture in the signature. On virtual threads the semaphore form is usually enough, since the "thread" being held is cheap; what remains scarce is whatever the call holds downstream, which the semaphore still bounds.

Load shedding at the edge is the same semaphore one level up: a filter that tryAcquires a permit per request and returns 503 immediately when none is available, sized from measured capacity (throughput × latency at the knee of the curve). Adaptive limiters replace the fixed size with a gradient algorithm: track the minimum observed latency, compare the current latency to it, and shrink the limit when latency rises (queueing has begun) and grow it slowly when it falls. Netflix's concurrency-limits and Envoy's adaptive concurrency filter both do this; the effect is a service that sits at the top of its throughput curve instead of over the cliff behind it.

#Walkthrough: the breaker that never opened, and the one that never closed

Two breakers in one incident.

resilience4j.circuitbreaker.instances:
  payments:
    slidingWindowSize: 100
    minimumNumberOfCalls: 100          # a quiet endpoint: 20 calls/min
    failureRateThreshold: 50
  search:
    slidingWindowSize: 10
    waitDurationInOpenState: 60s
    permittedNumberOfCallsInHalfOpenState: 1

1. The payments provider went down. payments received twenty calls a minute, every one a timeout at 5 s. The window needed 100 calls before it would judge: five minutes of every payment call hanging for five seconds, with the fallback never engaged, because the breaker was CLOSED and had not yet seen enough. Users saw spinners and the thread pool absorbed 100 stuck calls before anyone noticed. minimumNumberOfCalls must be reachable within the time you are willing to be slow: for twenty calls a minute, ten is a thirty-second judgement.
2. On the same afternoon search had a two-second blip. Its window of ten filled with failures in a second, it opened, and stayed open for 60 s. Then one trial call was permitted; it happened to land during a garbage-collection pause on the search pod and failed; the breaker reopened for another 60 s. Then another single trial, another unlucky failure. Search was healthy for the entire hour it showed "unavailable", because one trial call at a time is not a sample, and sixty seconds between samples is an eternity for a service that recovers in two.
3. The fixes were symmetric. payments: minimumNumberOfCalls: 10, slowCallDurationThreshold: 2s so timeouts registered as slow before they registered as failures, and a waitDurationInOpenState of 15 s. search: permittedNumberOfCallsInHalfOpenState: 10 so the judgement was a sample, waitDurationInOpenState: 5s, and automaticTransitionFromOpenToHalfOpenEnabled: true so a quiet period could not leave it open forever.
4. Both got the alert this lesson's PRODUCTION callout describes: open for longer than three wait durations pages someone, because that is a breaker hiding an outage behind a fallback.
5. The dashboards were changed to show resilience4j.circuitbreaker.state per instance as a timeline, so a breaker that flaps and one that sticks look different at a glance.

A breaker's parameters are a statement about the dependency's traffic and recovery time. Defaults tuned for a busy endpoint never open on a quiet one, and a tiny half-open sample turns one unlucky call into an hour of fallback.

#Try it yourself

#Misconceptions

• "A breaker opens after N failures." It opens when the failure or slow-call rate over a window crosses a threshold, and only once the window holds minimumNumberOfCalls. On a quiet endpoint that can take minutes.
• "An open breaker re-tests itself." It moves to half-open on the next call after the wait, or on a schedule only if automatic transition is enabled. No traffic, no test.
• "One half-open trial is enough." One call is not a sample; an unlucky one reopens the breaker for another full wait. Permit several.
• "A semaphore bulkhead isolates the dependency's threads." It caps how many caller threads can be inside the call; only the thread-pool form gives the dependency its own threads.
• "Load shedding is what the breaker does." Breakers protect the caller from dependencies; shedding protects the service from callers. Different direction, same semaphore.

#Going deeper

• Resilience4j documentation: CircuitBreaker (the state diagram and every parameter), Bulkhead, and the Micrometer metrics section.
• CircuitBreakerStateMachine and SlidingWindowMetrics in the Resilience4j source, a few hundred readable lines.
• Michael Nygard, Release It! (2nd ed.), chapters on stability patterns: Circuit Breaker, Bulkheads, Fail Fast, and the anti-patterns they answer.
• Netflix concurrency-limits README and Envoy's "Adaptive Concurrency" filter, for gradient-based shedding.
• Google SRE book, chapter 21, "Handling Overload", on load shedding and criticality.