The embedded server and its threads

A Boot service handles a request on a thread it borrowed from Tomcat, and that thread is the scarcest thing in the process. There are 200 of them by default. When a downstream slows down, every one of the 200 ends up waiting on it, the service stops answering health checks, the orchestrator restarts it, and the restart makes the downstream worse. Understanding the embedded server means understanding where requests queue, which knobs bound the queues, and how the process gets out of the way when it is told to stop.

#Tomcat, Jetty, and the switch

spring-boot-starter-web brings Tomcat. Exclude spring-boot-starter-tomcat and add spring-boot-starter-jetty to swap it; the application code does not change, and server.* properties that are generic keep working while the server.tomcat.* ones become server.jetty.*. Tomcat is the default because it is the most exercised path in Boot and because most operations knowledge on the internet assumes it. Change it for a reason you can state, such as an organisation-wide Jetty tuning practice, not for a benchmark.

#The path of a request

client ──TCP──▶ acceptor thread ──▶ poller (NIO selector) ──▶ worker thread pool ──▶ your controller
                     │                        │                       │
               accept-count            max-connections          threads.max
               (OS backlog)          (open sockets held)        (concurrent requests)

Three limits, and the order matters:

A request whose socket is accepted but for which no worker is free waits in the poller with the connection open; the client sees latency, not an error. Only when connections exceed 8192 and the backlog of 100 is also full does the client get a refused connection. So a saturated service looks slow long before it looks down, and the latency percentiles, not the error rate, are the first signal.

#Sizing the worker pool

200 threads sounds like capacity. It is only capacity if what those threads wait on can serve 200 at once. The default Hikari pool has 10 connections, so a service where every request touches the database has 10 requests working and 190 waiting for a connection, each holding a Tomcat thread and a socket. Size the pieces together:

• Little's law: concurrency = throughput × latency. 500 requests/s at 40 ms means about 20 requests in flight; 200 threads is plenty and 10 connections is the real ceiling.
• Raise threads.max only when the threads would have something to do. If they would wait on a pool or a downstream, raise that first or accept the limit.
• Lower threads.max on purpose when memory is tight: each thread has a stack (1 MB reserved by default) and a busy pool means a busy heap.

#Timeouts

Tomcat does not time out a request that is being processed. server.tomcat.connection-timeout is the time it waits for a client to send the request line and headers after connecting; server.tomcat.keep-alive-timeout is how long an idle keep-alive connection is held. Neither interrupts your controller. A controller that calls a downstream with no timeout holds its worker forever, and forever is how long it takes for 200 of them to accumulate.

So request-level timeouts are set on the outbound side, on every client the service uses:

@Bean
RestClient paymentsClient(RestClient.Builder builder) {
    var factory = new JdkClientHttpRequestFactory();
    factory.setReadTimeout(Duration.ofSeconds(2));                // per request
    return builder.baseUrl("https://pay.internal").requestFactory(factory).build();
}

spring.datasource.hikari.connection-timeout: 3000     # ms to wait for a connection from the pool
spring.datasource.hikari.max-lifetime: 1800000
spring.mvc.async.request-timeout: 30s                  # async controllers only

The rule: every blocking call a request makes has a timeout shorter than the caller's patience, and the sum along the longest path is shorter than the load balancer's timeout. A service that answers in 31 seconds to a balancer that gave up at 30 did the work and got the error.

#Graceful shutdown

By default, stopping the JVM drops in-flight requests on the floor; the client gets a reset. Boot's graceful shutdown lets them finish:

server.shutdown: graceful
spring.lifecycle.timeout-per-shutdown-phase: 30s     # default 30s

On SIGTERM, Tomcat stops accepting new connections, the requests already in the workers run to completion, and the context closes once they finish or the timeout passes. Boot also flips its readiness state to REFUSING_TRAFFIC at the start of shutdown, so /actuator/health/readiness reports out of service and the platform stops routing.

That last step has a race that the platform has to cover. Kubernetes sends SIGTERM and removes the pod from the Service endpoints concurrently; for a second or two, new requests still arrive at a pod that has stopped accepting. The fix is a preStop hook that sleeps a few seconds so the endpoint removal propagates before the signal lands, and a terminationGracePeriodSeconds longer than Boot's timeout plus that sleep. Graceful shutdown in Boot without the preStop delay still produces a burst of connection errors on every deploy.

#The virtual threads switch

On Java 21 or later, one property changes the model:

spring.threads.virtual.enabled: true

Tomcat then runs each request on a virtual thread instead of a pooled platform thread, and Boot switches its own executors too: @Async, @Scheduled, and the Kafka and RabbitMQ listener containers. threads.max stops being the concurrency limit; virtual threads are cheap enough that the limit moves to whatever the requests wait on, which for most services means the connection pool becomes the ceiling immediately and visibly. That is the honest state of affairs the platform-thread pool was hiding.

Two things to check before flipping it in production. Code that holds a synchronized block across a blocking call pins the carrier thread on JDKs before 24, which reduces the benefit and can deadlock a small carrier pool; -Djdk.tracePinnedThreads finds those. And thread-local-heavy libraries (some tracing and security context holders) now create one context per request, which is correct but costs memory that was previously shared across 200 threads.

#Under the hood: the NIO connector, thread by thread

Embedded Tomcat is the same Tomcat as the standalone one, started programmatically by TomcatServletWebServerFactory, with one Connector on server.port using the NIO protocol handler (Http11NioProtocol). Its threads have distinct jobs. One acceptor thread sits in ServerSocketChannel.accept(); each accepted socket is registered with the poller, a single thread running a Selector loop over every open connection. When a socket has bytes to read, the poller hands it to the worker executor, a ThreadPoolExecutor sized by threads.min-spare/threads.max, and a worker parses the request, runs the servlet filter chain and DispatcherServlet, writes the response, and returns the socket to the poller to wait for the next request on that keep-alive connection. max-connections is a LimitLatch the acceptor takes before accepting; when it is exhausted the acceptor blocks, and the kernel's listen backlog (accept-count, the backlog argument to listen(2)) fills behind it.

Two details explain most tuning surprises. The worker executor is Tomcat's own StandardThreadPool, which, unlike the JDK's, grows to threads.max before it queues (it prefers creating a thread over queuing until the max), so min-spare is a floor and not a target. And the servlet request is synchronous on the worker by default: DeferredResult/Callable controllers and WebFlux hand the socket back to the poller while the work happens elsewhere, which is what spring.mvc.async.request-timeout bounds. Under spring.threads.virtual.enabled=true, Boot swaps the worker executor for a virtual-thread-per-task executor via a TomcatProtocolHandlerCustomizer; the acceptor and poller stay platform threads, and threads.max is no longer consulted.

Graceful shutdown is GracefulShutdown in the Boot Tomcat module: it pauses the connector (the acceptor stops taking connections, the LimitLatch is released), waits for the worker executor's active count to reach zero or the phase timeout, then closes. Requests already inside a worker finish; requests sitting in the kernel backlog are reset when the socket closes, which is the population the Kubernetes preStop sleep is protecting.

#Walkthrough: the restart storm with a 10-connection pool

A catalogue service with defaults everywhere: 200 workers, Hikari at 10, no outbound timeouts.

server.tomcat.threads.max: 200        # default
spring.datasource.hikari.maximum-pool-size: 10   # default
spring.datasource.hikari.connection-timeout: 30000   # default, 30 s

1. A slow query appeared after a data growth; p99 on one endpoint went from 20 ms to 4 s. Ten workers held the ten connections for four seconds each; the other 190 workers queued in Hikari for a connection, each holding a Tomcat worker and its socket.
2. After 30 s, Hikari threw SQLTransientConnectionException for the oldest waiters, so the service returned 500s at a slow rate while its latency for every endpoint, including ones that touched no database, climbed past the load balancer's 10 s timeout, because there were no free workers to run them.
3. The liveness probe was /actuator/health, which included the datasource indicator, which needed a connection, which waited 30 s. The probe timed out at 5 s; after three failures Kubernetes restarted the pod. Every pod, in a wave, since they were all in the same state.
4. Each restart dropped its in-flight requests (no graceful shutdown), reconnected to the database (ten new connections and their handshakes), and rebuilt its caches with cold queries, which made the slow query slower for the pods that were still up.
5. The fixes, applied in order of impact: the liveness probe moved to /actuator/health/liveness (application state only); connection-timeout cut to 3 s so a waiter fails fast and frees its worker; threads.max lowered to 50, since ten connections could not keep 200 busy and a shorter queue means a shorter wait; the query indexed; server.shutdown=graceful with a 5 s preStop. The Tomcat busy-threads graph, once the MBean registry was enabled, showed the saturation ten minutes before the first alert.

Every layer had a default that was individually reasonable and collectively a restart loop. The pool, not the thread count, was the capacity; the health check was the trigger; the missing timeouts were the fuel.

#Try it yourself

#Misconceptions

• "Tomcat times out slow requests." It times out idle connections and slow request headers. A request inside a worker runs until your code returns; only outbound timeouts bound it.
• "More worker threads means more capacity." Only if what they wait on can serve more. Threads beyond the pool or downstream capacity are a longer queue.
• "max-connections is the concurrency limit." It is the count of open sockets, mostly idle keep-alives handled by one poller. threads.max is the concurrency limit.
• "A saturated service returns errors." It returns latency, for thousands of requests, before the kernel backlog overflows. Alert on busy threads and latency.
• "Graceful shutdown alone makes deploys clean." The platform must stop routing before the signal, and wait long enough after. That is preStop and the grace period.

#Going deeper

• Spring Boot reference, "Embedded Web Servers", including "Graceful Shutdown" and the server.tomcat.* property list.
• Tomcat documentation, "The HTTP Connector", the maxThreads, maxConnections and acceptCount sections, and NioEndpoint source for the acceptor/poller/worker split.
• HikariCP wiki, "About Pool Sizing", the argument for small pools.
• Kubernetes documentation, "Container Lifecycle Hooks" and "Pod termination", for the endpoint-removal race.
• Spring Framework reference, "Asynchronous Requests", for what returns the socket to the poller.