Executors and thread pools

Creating a thread per task worked for a few dozen tasks and fell apart at a few thousand, so Java gave us the pool: a fixed set of workers pulling tasks from a queue. ThreadPoolExecutor has seven constructor parameters, and every production incident involving a pool traces back to one of them being left at a default that was wrong for the workload. This lesson is those parameters, what a worker thread actually does all day, the two sizing rules, the exception that disappears, and the out-of-memory error that arrives an hour after the traffic did.

#The executor abstraction

ExecutorService pool = Executors.newFixedThreadPool(8);
Future<Report> f = pool.submit(() -> buildReport(id));     // Callable → Future
pool.execute(() -> audit(event));                           // Runnable, fire and forget
Report r = f.get(5, TimeUnit.SECONDS);                      // wait, with a timeout
pool.shutdown();                                            // stop accepting; finish queued work
pool.awaitTermination(30, TimeUnit.SECONDS);

Submitting a task establishes happens-before between the submitter's prior writes and the task; Future.get establishes it between the task's writes and the caller. The pool is how work crosses threads safely.

#ThreadPoolExecutor's seven parameters

new ThreadPoolExecutor(
    corePoolSize,        // threads kept alive even when idle
    maximumPoolSize,     // ceiling, reached only when the queue is FULL
    keepAliveTime, unit, // how long threads above core live idle
    workQueue,           // where tasks wait
    threadFactory,       // names threads; sets daemon flag and uncaught handler
    rejectionHandler     // what happens when queue and pool are both full
);

The interaction people get wrong: a new task goes to a core thread if one is free; otherwise it goes to the queue; only if the queue is full does the pool create a thread up to maximum; only if the pool is at maximum too does the rejection handler run. So with an unbounded queue, maximumPoolSize is never reached — the queue absorbs everything — and the pool never grows past core. This is what Executors.newFixedThreadPool(n) builds: n threads, unbounded LinkedBlockingQueue.

#Under the hood: what a worker does

A ThreadPoolExecutor keeps its state in one AtomicInteger called ctl: the top three bits are the run state (RUNNING, SHUTDOWN, STOP, TIDYING, TERMINATED) and the rest is the worker count, so both change in one CAS. Each worker is a thread running one loop:

Runnable task = firstTask;
while (task != null || (task = getTask()) != null) {   // getTask() blocks on queue.take(), or poll(keepAlive)
    try { task.run(); }
    catch (Throwable t) { thrown = t; throw t; }         // the worker dies; processWorkerExit replaces it
    finally { task = null; completedTasks++; }
}

getTask() is where a thread dump shows idle workers: WAITING (parking) in LinkedBlockingQueue.take. A worker above corePoolSize uses poll(keepAliveTime) instead and exits when it times out. submit wraps the callable in a FutureTask before it reaches the queue, which is the object that swallows the exception below; execute puts the raw Runnable in.

The other pool worth knowing is ForkJoinPool: each worker has its own deque, pushes subtasks to it, and steals from the tail of other workers' deques when its own is empty. That is what parallelStream() and CompletableFuture.supplyAsync() without an executor run on, through the shared commonPool(), sized to cores minus one. It is built for many small CPU-bound tasks that split; a blocking I/O call on it holds one of a handful of workers.

#The unbounded queue

An unbounded queue never rejects. When producers outrun consumers, the queue grows without limit — each queued task holding its captured arguments — until the heap is exhausted. The failure is an OutOfMemoryError an hour after the traffic spike, with a heap dump full of FutureTasks. The pool did exactly what it was told.

Bound the queue, and choose what happens when it fills:

CallerRunsPolicy is the one to reach for in a service: when the pool is saturated, the request thread does the work, which slows it down, which slows the rate of new submissions. Rejection is a feature; it is the pool telling you it is full.

#Walkthrough: the OutOfMemoryError at 03:10

An order service published an audit event per order to a pool: Executors.newFixedThreadPool(4), each task serialising the order and POSTing it to an audit API. At 02:00 the audit API's latency went from 20 ms to 2 s. Trace the numbers:

1. Four workers, each now completing a task every 2 s: 2 tasks/s drained. Orders arrive at 60/s. The queue grows by 58 tasks per second.
2. Each queued FutureTask holds a lambda that captured the Order object: about 4 KB with its lines. 58 × 4 KB = 230 KB/s, 14 MB/min, 830 MB/hour.
3. At 03:10 the heap (1 GB) is full of FutureTasks. OutOfMemoryError: Java heap space on a random request thread, whose allocation happened to be the one that failed. The stack trace points at JSON parsing in the checkout endpoint, which has nothing to do with audit.
4. Nothing was logged at 02:00, because nothing failed: submit returned instantly every time, as an unbounded queue guarantees.
5. The fix: new ArrayBlockingQueue<>(1_000) and CallerRunsPolicy. At 02:00 the queue fills in 17 seconds; from then on request threads POST the audit event themselves at 2 s each, checkout latency climbs, the latency alert fires at 02:01, and the on-call engineer sees "audit API slow", which is the truth.

An unbounded queue converts a dependency's latency problem into your memory problem, an hour later, with a misleading stack trace. A bounded one converts it into your latency problem, immediately, with the right name on it.

#Sizing

Two workloads, two answers:

• CPU-bound (compression, parsing, computation): more threads than cores just adds context switching. Size ≈ number of cores. Runtime.getRuntime().availableProcessors() — which is container-aware since Java 10, so a pod with a 2-CPU limit reports 2 (and a fractional limit like 1.5 rounds up to 2).
• I/O-bound (database calls, HTTP): threads spend most of their time waiting. Size ≈ cores × (1 + wait time / compute time). A thread that waits 45 ms and computes 5 ms could be one of ten per core. Or, since Java 21, use virtual threads and stop sizing.

Then look at what the threads wait on. Fifty threads calling a database through a ten-connection pool means forty threads waiting for a connection. The bottleneck moves; size the pieces together. And do not use newCachedThreadPool in a service: it has maximumPoolSize = Integer.MAX_VALUE and a SynchronousQueue, so every submission that finds no idle thread creates one, and a slow dependency turns into ten thousand threads and an OutOfMemoryError: unable to create native thread.

#The exception that disappears

pool.submit(() -> {
    process(order);                 // throws NullPointerException
});
// nothing is logged. Nothing. The Future holds it; nobody calls get().

submit wraps the task in a FutureTask, which catches any exception and stores it for get() to rethrow. If nobody calls get(), the failure is invisible. execute (not submit) lets the exception propagate out of task.run() in the worker loop above, to the thread's UncaughtExceptionHandler, which prints it by default and kills the worker (the pool replaces it). Rules:

• Use execute for fire-and-forget so failures surface — and set an UncaughtExceptionHandler in the thread factory that logs properly.
• If you submit, you must get() or otherwise inspect the Future.
• Or catch inside the task and log there. A task that can throw and is never observed is a task whose failures you have chosen not to know about.

#Shutdown

shutdown() stops accepting and lets queued tasks finish; shutdownNow() interrupts running tasks and returns the queue. Neither happens automatically. A pool with non-daemon threads keeps the JVM alive after main returns. In Spring, declare the executor as a bean so the container shuts it down; in Java 19+, ExecutorService is AutoCloseable and close() does shutdown-and-await.

#Naming threads

pool-3-thread-7 in a thread dump tells you nothing. A ThreadFactory that names threads report-worker-1 turns the dump into a map of your system. Spring's ThreadPoolTaskExecutor has setThreadNamePrefix. Do this on every pool; it costs one line.

#Scheduled executors

ScheduledThreadPoolExecutor runs tasks after a delay or at a fixed rate. scheduleAtFixedRate skips nothing and catches up if a run overruns (runs back to back, never concurrently); scheduleWithFixedDelay waits the delay after each run. And: an exception in a scheduled task cancels all future runs silently. The task's FutureTask completes exceptionally and is never re-queued. Wrap the body in a catch-all that logs, or the job that "stopped running last Tuesday" will be this.

#Try it yourself

#Misconceptions

• "maximumPoolSize is how many threads the pool can use." Only when the queue fills. With an unbounded queue it is never consulted; newFixedThreadPool sets it equal to core for that reason.
• "The pool logs task failures." submit stores them in the Future; execute hands them to the thread's handler. Neither logs unless you arranged it.
• "An idle pool costs nothing." Core threads are alive and parked; each holds a stack. Fifty pools of eight threads in one JVM is four hundred threads doing nothing.
• "newCachedThreadPool scales with load." It creates a thread per concurrent task with no ceiling; under a slow dependency that is thousands of threads and a native-thread OOM.
• "Scheduled tasks retry after an error." They stop, silently, forever.

#Going deeper

• ThreadPoolExecutor Javadoc, the class comment: the queuing rules, in the authors' words.
• ThreadPoolExecutor.runWorker and getTask source: the loop every pool thread runs.
• Brian Goetz, Java Concurrency in Practice, chapter 8, for the sizing formula and the saturation policies.
• Spring Boot's TaskExecutionAutoConfiguration and spring.task.execution.* properties.
• ForkJoinPool Javadoc for work stealing and ManagedBlocker.