Thread problems

Three thread problems account for most of the "the service is up but not answering" incidents in Java: a deadlock, a thread pool exhausted by work that never finishes, and contention on a lock that one slow holder keeps. They produce the same symptom — requests hang, CPU is low — and one tool tells them apart in under a minute: the thread dump.

Each dump in this lesson was taken with jcmd <pid> Thread.print from a small program running on Java 21 that reproduces the problem.

#Reading a thread dump

A dump lists every thread with its name, its state, and its stack. The states that matter:

Name your threads. A dump full of pool-7-thread-3 tells you nothing about which pool is exhausted. Every executor should get a thread factory with a meaningful prefix.

#Deadlock

Two transfers between the same two accounts, in opposite directions, each locking its source account first:

static void transfer(String name, Object from, Object to) {
    synchronized (from) {
        sleep(100);
        synchronized (to) { System.out.println(name + " done"); }
    }
}

transfer-1041 locks A and waits for B; transfer-1042 locks B and waits for A. Neither will ever finish. The JVM detects monitor deadlocks itself, and the dump says so at the end:

Found one Java-level deadlock:
=============================
"transfer-1041":
  waiting to lock monitor 0x0000ffff38001c10 (object 0x000000008bb14df0, a java.lang.Object),
  which is held by "transfer-1042"
 
"transfer-1042":
  waiting to lock monitor 0x0000ffff3c000f00 (object 0x000000008bb14de0, a java.lang.Object),
  which is held by "transfer-1041"
 
Java stack information for the threads listed above:
===================================================
"transfer-1041":
	at Threads.transfer(Threads.java:10)
	- waiting to lock <0x000000008bb14df0> (a java.lang.Object)
	- locked <0x000000008bb14de0> (a java.lang.Object)
"transfer-1042":
	at Threads.transfer(Threads.java:10)
	- waiting to lock <0x000000008bb14de0> (a java.lang.Object)
	- locked <0x000000008bb14df0> (a java.lang.Object)
 
Found 1 deadlock.

Everything is there: the two threads, the object each holds (locked <…de0>, locked <…df0>), the object each wants, and the line where they are stuck. The cycle is visible by matching the addresses.

The fix is structural: acquire locks in a consistent order — for example, always lock the account with the lower id first, whichever direction the money moves. With ReentrantLock, tryLock with a timeout turns a permanent hang into a failure you can retry. The JVM's detector also finds deadlocks between ReentrantLocks, which it reports as "ownable synchronizers"; it cannot see a deadlock spread across two processes or two database transactions, which the database reports on its own side.

#Pool exhaustion

A request pool of four threads. Each request calls a pricing service with no timeout, and the pricing service has stopped answering. Two hundred requests arrive:

queued behind the pool: 196

Four requests are being "processed" and 196 are waiting for a thread. Every thread in the pool is in the same place:

"http-nio-8080-exec-1" #20 [94] prio=5 os_prio=0 cpu=0.26ms elapsed=2.17s tid=0x0000ffffa037f950 nid=94 waiting on condition  [0x0000ffff3fffd000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21.0.12/Native Method)
	- parking to wait for  <0x000000008bb14ff8> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(java.base@21.0.12/LockSupport.java:221)
	...
	at java.util.concurrent.CountDownLatch.await(java.base@21.0.12/CountDownLatch.java:230)
	at Threads.callPricingService(Threads.java:17)

threads parked in callPricingService: 4

The diagnosis is the count: every thread of one pool parked in the same frame. cpu=0.26ms confirms none of them is doing work. In a real service the frame would be a socket read inside an HTTP client or a JDBC driver — the thread's state is often RUNNABLE there, because the JVM counts a blocking native read as running, so read the frame, not only the state.

The CPU is idle, the process is healthy, the health check may still pass, and the service is completely unavailable. The fixes, in order of importance:

• A timeout on every outbound call — connect timeout and read timeout — so a dead dependency returns threads within seconds.
• A bulkhead: a separate, bounded pool for each dependency, so a failing pricing service exhausts the pricing pool rather than the request threads that also serve every other endpoint.
• A circuit breaker that stops calling a dependency that is timing out, and fails fast instead.
• A bounded queue in front of the pool, so excess work is rejected rather than accumulated — the backpressure lesson's point.

The database connection pool is the same problem one layer down. The slow APIs lesson shows its evidence from the pool's side.

#Lock contention

Six worker threads write to an audit log guarded by one synchronized lock, and one write takes five seconds while holding it:

"order-worker-1" #20 [142] ... waiting on condition
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(java.base@21.0.12/Thread.java:509)
	at Threads.sleep(Threads.java:13)
	at Threads.writeAudit(Threads.java:20)
"order-worker-2" #21 [143] ... waiting for monitor entry
   java.lang.Thread.State: BLOCKED (on object monitor)
	at Threads.writeAudit(Threads.java:20)
	- waiting to lock <0x000000008bb15030> (a java.lang.Object)
"order-worker-3" #22 [144] ... waiting for monitor entry
   java.lang.Thread.State: BLOCKED (on object monitor)
	at Threads.writeAudit(Threads.java:20)
	- waiting to lock <0x000000008bb15030> (a java.lang.Object)

And the whole dump summarised by state:

      5    java.lang.Thread.State: BLOCKED (on object monitor)
      8    java.lang.Thread.State: RUNNABLE
      1    java.lang.Thread.State: TIMED_WAITING (parking)
      2    java.lang.Thread.State: TIMED_WAITING (sleeping)
      1    java.lang.Thread.State: WAITING (on object monitor)

Five threads BLOCKED waiting for the same object, <0x000000008bb15030>, and one thread holding it while it sleeps. That is not a deadlock — it will clear when the holder finishes — but it serialises six threads' work behind the slowest one. Counting states with grep | sort | uniq -c is a quick first look at any dump; a large BLOCKED count on one address is the finding.

The fix is almost always to do less inside the lock: hold it for the in-memory state change only, and move I/O — the write, the network call, the sleep before a retry — outside it. Where a single writer is really required, hand the work to a queue and one dedicated thread, so callers never wait for the I/O at all.

#Taking dumps in production

• Take three, a few seconds apart. A thread in the same frame every time is stuck; one in different frames is busy. One dump cannot tell the difference.
• jcmd <pid> Thread.print works on a running JVM without stopping it for more than a moment. kill -3 <pid> writes the same dump to the process's standard output, which is useful when jcmd is not in the container image.
• Spring Boot Actuator's /actuator/threaddump returns the same information over HTTP, and should be exposed only on a management port that is not public.
• Capture before restarting. A restart makes the symptom go away and takes every thread's stack with it.