Diagnosing concurrency bugs

Concurrency bugs do not reproduce on demand. They appear under load, on the third Tuesday, in production, and the log shows nothing because nothing threw. What you have instead is a thread dump: a snapshot of every thread, what it is doing, and what it is waiting for. Reading one is the skill this lesson teaches — first for the four classic shapes, then for making the bug reproduce on purpose so you can prove it fixed. Along the way: what the JVM has to do to take a dump, which is why a dump sometimes takes seconds and why one thread can delay all the others.

#Taking a thread dump

jcmd <pid> Thread.print > dump.txt                     # preferred; platform threads
jcmd <pid> Thread.dump_to_file -format=json dump.json  # Java 21+: includes virtual threads
jstack <pid> > dump.txt                                # older, same output as Thread.print
kill -3 <pid>                                          # prints to the JVM's stdout, no tools needed

In a container: kubectl exec <pod> -- jcmd 1 Thread.print. Take three dumps, ten seconds apart. One dump shows what threads are doing; three show what they are stuck doing — a thread in the same frame across all three is the one to look at. Spring Boot Actuator's /actuator/threaddump returns the same data as JSON.

#Under the hood: safepoints, and why a dump can take seconds

A thread dump needs every thread's stack to be stable while it is walked, so the JVM brings all threads to a safepoint: a point where the thread's state (its frames, its registers holding references) is fully described by metadata the JVM can read. Compiled code polls a flag at method returns and loop back-edges; when the VM requests a safepoint, every thread stops at its next poll and waits. Interpreted code and blocked threads are always at a safepoint.

Two consequences you will meet. A thread in a long-running counted loop without a poll (the JIT may elide polls from int-indexed loops it can prove finite; -XX:+UseCountedLoopSafepoints keeps them, and is the default since Java 10) can delay the safepoint for everyone; -Xlog:safepoint shows "time to safepoint" and names the slow thread. And the same mechanism serves garbage collection, deoptimisation and jcmd, so a dump taken during a GC pause waits for the pause. A dump is a stop-the-world event; three ten-second-apart dumps on a healthy service cost tens of milliseconds each, which is fine, and an automated dump every second is not.

What the dump prints per thread comes from those stable frames: the - locked <0x…> and - waiting to lock <0x…> annotations are the monitor state read from the object headers and the thread's lock records, and the Locked ownable synchronizers section at the end of each thread lists the AQS locks (ReentrantLock, Semaphore permits) it holds, which is the only way to find a ReentrantLock owner, since AQS parking shows as WAITING, not BLOCKED.

#Reading one thread

"http-nio-8080-exec-17" #48 daemon prio=5 os_prio=0 cpu=1203.44ms elapsed=8812.30s tid=0x... nid=0x2f1a waiting on condition [0x00007f...]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
        - parking to wait for <0x00000006c1a3f2b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(...)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(...)
        at com.zaxxer.hikari.util.ConcurrentBag.borrow(ConcurrentBag.java:151)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:180)
        ...
        at com.shop.billing.InvoiceRepository.findByCustomer(InvoiceRepository.java:44)

Name (set your thread names), state, and a stack. This one is a Tomcat request thread, TIMED_WAITING, parked inside HikariCP's borrow — it is waiting for a database connection. The first frame in your package tells you which code path asked for it. cpu=1203.44ms is the thread's total CPU time; comparing it across three dumps tells you whether the thread is working or waiting.

States: RUNNABLE (running or ready — including threads blocked in native I/O, which show as RUNNABLE because the JVM does not see inside the system call), BLOCKED (waiting for a monitor), WAITING/TIMED_WAITING (park, wait, sleep, join).

#Shape 1: deadlock

Found one Java-level deadlock:
=============================
"transfer-worker-2":
  waiting to lock monitor 0x... (object 0x000000076b1a2c10, a com.shop.Account),
  which is held by "transfer-worker-5"
"transfer-worker-5":
  waiting to lock monitor 0x... (object 0x000000076b1a2d48, a com.shop.Account),
  which is held by "transfer-worker-2"
 
Java stack information for the threads listed above:
"transfer-worker-2":
        at com.shop.Ledger.transfer(Ledger.java:31)
        - waiting to lock <0x000000076b1a2c10> (a com.shop.Account)
        - locked <0x000000076b1a2d48> (a com.shop.Account)

The JVM detects monitor and ReentrantLock deadlocks and prints the cycle at the bottom of the dump. Two threads, each holding one account and waiting for the other: transfer(a, b) and transfer(b, a) ran at once. Symptom in production: a subset of requests hang forever, the threads never return to the pool, and the count of BLOCKED threads grows until the pool is gone.

Fix: acquire locks in a global order — sort by account id, lock the lower first. Or use tryLock with a timeout so the cycle breaks. Then write the test below.

#Shape 2: thread-pool exhaustion

No deadlock section. Instead, all 200 http-nio-*-exec-* threads look like the HikariCP example — every one parked in the same frame, waiting on the same resource. Or every one inside HttpClient.send to the same host. The pool is not deadlocked; it is full of threads waiting on something slow.

Symptom: latency climbs to the timeout, then requests fail with 503 or connection refused; CPU is near idle. Cause: a slow downstream plus no timeout, a connection leak, a query that stopped using its index, a lock held during I/O.

Read: which frame are they all in? That is the resource. Then: why is it slow or unavailable? Hikari's pending count, the downstream's latency, the database's active queries. The fix is a timeout and a bulkhead so one slow dependency cannot absorb every thread, then the root cause.

#Walkthrough: three dumps, one grep

The pager says "checkout p99 > 10 s". Three dumps, ten seconds apart. Do not read them top to bottom; count:

$ for f in dump1 dump2 dump3; do echo "== $f"; grep -A3 '"http-nio' $f | grep 'State:' | sort | uniq -c; done
== dump1
    187 java.lang.Thread.State: TIMED_WAITING (parking)
     11 java.lang.Thread.State: RUNNABLE
      2 java.lang.Thread.State: WAITING (parking)
== dump2
    194 java.lang.Thread.State: TIMED_WAITING (parking)
      6 java.lang.Thread.State: RUNNABLE
== dump3
    196 java.lang.Thread.State: TIMED_WAITING (parking)
      4 java.lang.Thread.State: RUNNABLE

$ grep -A12 '"http-nio' dump3 | grep -E '^\s+at com\.' | sort | uniq -c | sort -rn | head -3
    189 at com.shop.pricing.RateClient.current(RateClient.java:58)
      4 at com.shop.checkout.CheckoutService.place(CheckoutService.java:112)
      3 at com.shop.cart.CartRepository.load(CartRepository.java:40)

1. 196 of 200 request threads are parked, and the number is rising across dumps: the pool is draining, not deadlocked (no cycle printed).
2. 189 of them are in RateClient.current, line 58. That is the resource. The frames above it in the dump are HttpClient.send and a socket read: they are waiting on the rates service's response.
3. RateClient has no read timeout (line 58 is a send with the default), so a rates service that hung at 10:41 held every thread that touched it, and checkout touches it once per request.
4. The rates service was returning in 12 s, not never. With a 2 s timeout the checkout would have failed fast at 10:41 with a clear error; with a Semaphore(20) bulkhead, 180 threads would have been free to serve cart and catalogue, which do not need rates at all.
5. Fix in that order: timeout, bulkhead, then find out why rates was slow, which is someone else's incident.

Ten minutes, two greps. The diagnosis was in the count, not in any single stack.

#Shape 3: a hot loop

One or two threads RUNNABLE, in the same frames across all three dumps, and cpu= in their header climbing by thousands of milliseconds between dumps. top -H -p <pid> shows which OS thread is burning; its id in hex matches nid= in the dump.

Causes: an infinite loop on a HashMap corrupted by unsynchronised writes (classic pre-Java-8, still possible); a retry loop with no backoff; a busy-wait on a flag that is not volatile (which the JIT hoisted — the thread will never see the change); a regex with catastrophic backtracking. The frame names it.

#Shape 4: the lost update

No dump helps. The service is fine; the numbers are wrong. Two requests read a balance of 100, both subtract 30, both write 70. Or a HashMap used as a cache from several threads silently loses entries. Or a counter is short by a few percent. These are data races, and they leave no trace except wrong data.

Find them by reading: every field written from a request thread and read from another — is it under a lock, volatile, atomic, or a concurrent collection? HashMap, ArrayList, SimpleDateFormat, a non-final field in a @Service singleton: each is a candidate. Static analysis (SpotBugs' IS2_INCONSISTENT_SYNC, Error Prone's GuardedBy) finds a share of them. The rest, you find by thinking about which state is shared.

#Beyond dumps: JFR and profilers

A dump is a snapshot; Java Flight Recorder is a film. jcmd <pid> JFR.start duration=120s filename=rec.jfr records, with under 2% overhead, the events that name concurrency problems: jdk.JavaMonitorEnter (every contended monitor acquisition over 20 ms, with the lock's class and the waiting stack), jdk.ThreadPark (every long park, with what it parked on), jdk.ExecutionSample (a sampling profile of what threads are doing), and jdk.VirtualThreadPinned. Open it in JDK Mission Control and the "Lock Instances" view lists the hottest locks in the JVM, sorted, which is the answer to "what is contended" that no dump can give. For the hot-loop shape, async-profiler in CPU mode gives a flame graph, and in lock mode it profiles contention with less overhead than JFR.

#Making it reproduce

A concurrency bug you cannot reproduce is one you cannot prove fixed. The test forces the interleaving:

@Test
void transferInBothDirectionsDoesNotDeadlock() throws Exception {
    Ledger ledger = new Ledger();
    Account a = ledger.open(100), b = ledger.open(100);
    CountDownLatch start = new CountDownLatch(1);
    ExecutorService pool = Executors.newFixedThreadPool(2);
 
    Future<?> f1 = pool.submit(() -> { start.await(); for (int i = 0; i < 10_000; i++) ledger.transfer(a, b, 1); return null; });
    Future<?> f2 = pool.submit(() -> { start.await(); for (int i = 0; i < 10_000; i++) ledger.transfer(b, a, 1); return null; });
    start.countDown();                                   // release both at the same instant
 
    f1.get(10, TimeUnit.SECONDS);                        // a deadlock fails here with TimeoutException
    f2.get(10, TimeUnit.SECONDS);
    assertEquals(200, a.balance() + b.balance());        // a lost update fails here
}

The latch makes both threads start together; the loop gives the race ten thousand chances; the timeout turns a hang into a failure; the invariant assertion catches lost updates. Run it against the broken code first — a test that has never failed has not proven anything. Then fix, then run it a hundred times.

For subtler interleavings, jcstress (the JDK's own concurrency stress harness) runs millions of trials and reports every outcome observed, including the ones the memory model allows but you did not expect.

#Try it yourself

#Misconceptions

• "A dump shows what is slow." It shows what is waiting. Combine three dumps, or use JFR, to see what is slow.
• "RUNNABLE means running." A thread in a native socket read shows RUNNABLE. Check the frame, not the state.
• "If the JVM found no deadlock, there is no deadlock." It detects cycles between monitors and AQS locks. A cycle involving a database lock, a semaphore with no permits, or a CompletableFuture that never completes is invisible to it.
• "Taking dumps is free." Each one is a safepoint, a stop-the-world event. A few per incident is fine; one per second from a monitoring agent is a performance problem.
• "A passing test proves the race is fixed." It proves it is less likely. The failing run before the fix, plus reasoning about the locks, is the proof; jcstress is the confirmation.

#Going deeper

• jcmd documentation: Thread.print, Thread.dump_to_file, JFR.start, VM.flags.
• Aleksey Shipilëv, "JVM Anatomy Quark #22: Safepoint polls", and Nitsan Wakart, "Safepoints: Meaning, Side Effects and Overheads".
• JDK Mission Control's user guide, the "Lock Instances" and "Thread Dump" pages.
• jcstress: the samples directory, and the @Outcome annotation that documents which interleavings are acceptable.
• fastthread.io and similar dump analysers, for a first pass on a 2,000-thread dump.