Finding memory leaks

In a garbage-collected language, a memory leak is not memory you forgot to free. It is memory you still reference and will never use again. The collector is doing exactly its job — keeping everything reachable — and the bug is a reference path nobody meant to keep.

So finding a leak is finding that path. The production engineering course's lesson on memory leaks covers recognising one from the outside — the old-generation floor that only rises. This lesson is about the inside: the tools that show what is accumulating and what is holding it, applied to a leak that heap histograms alone make hard to see.

#The leak: a ThreadLocal on a pool

A service collects the audit events of the current request in a thread-local list, and flushes them at the end:

static final ThreadLocal<List<AuditEvent>> PENDING = ThreadLocal.withInitial(ArrayList::new);
 
static void handleRequest(int n) {
    PENDING.get().add(new AuditEvent("order-" + n, "PRICE_CHECKED"));
    flush();
}
static void flush() {
    for (AuditEvent e : PENDING.get()) { /* write somewhere */ }
    // missing: PENDING.get().clear()  — or better, PENDING.remove() in a finally
}

In a thread-per-request world, the list would die with the thread. But requests run on a pool of eight threads that live as long as the application, so each thread's list keeps every event it ever handled. Used heap after a full collection, every 16,000 requests:

after 16,000 requests: 21 MB used after GC
after 32,000 requests: 38 MB used after GC
after 48,000 requests: 55 MB used after GC
after 64,000 requests: 73 MB used after GC
after 80,000 requests: 90 MB used after GC

About 17 MB per 16,000 requests, forever. At a thousand requests a second, that is roughly a gigabyte an hour.

#Step one: what is accumulating

jcmd <pid> GC.class_histogram (or jmap -histo:live <pid>) lists live objects by class. For this leak it would show AuditEvent, byte[] and Object[] climbing between two histograms taken a few minutes apart — the classes to suspect are the ones whose counts only grow.

A histogram shows shallow size: the bytes of each object itself. An ArrayList is 24 bytes whether it holds nothing or ten thousand events. What matters for a leak is retained size: everything that would be freed if that object became unreachable. Heap analysers compute it from a dominator tree — object X dominates object Y if every path from a root to Y passes through X — and sorting by retained size puts the single object that holds the most memory at the top.

For this leak, the retained size can be measured directly: make each pool thread call PENDING.remove(), collect, and compare used heap before and after.

retained by the ThreadLocal lists: 89 MB (freed when each pool thread called remove())

Eight ArrayList objects — 192 bytes of shallow size between them — retained 89 MB.

#Step two: what is holding it

Knowing that AuditEvent objects accumulate does not say why. The answer is the path from a GC root to those objects, and there are two ways to get it.

A heap dump. jcmd <pid> GC.heap_dump /tmp/app.hprof writes every object to a file, which a heap analyser such as Eclipse Memory Analyzer (MAT) or VisualVM opens. "Path to GC roots" on a suspect object shows the reference chain; the dominator tree shows which single object retains the most. The cost: a full pause while the dump is written, a file as large as the live heap, and a file that contains every value in memory — tokens, personal data — so it must be handled like production data.

Java Flight Recorder's old object sample. JFR, built into the JDK, samples allocations and keeps track of the ones that stay alive. Dumping a recording with path-to-gc-roots=true makes it walk from each surviving sample back to its root. No separate tool, no multi-gigabyte file:

$ java -XX:StartFlightRecording=name=leak,settings=profile ...
$ jcmd <pid> JFR.dump name=leak filename=leak.jfr path-to-gc-roots=true
$ jfr print --events jdk.OldObjectSample leak.jfr

One of the 246 samples it recorded:

jdk.OldObjectSample {
  objectSize = 1.0 kB
  objectAge = 1.21 s
  lastKnownHeapUsage = 73.0 MB
  object =  [
    byte[1024]
    snapshot : AuditLeak$AuditEvent
    [9625] : java.lang.Object[14053]
    elementData : java.util.ArrayList Size: 10370
    value : java.lang.ThreadLocal$ThreadLocalMap$Entry
    [12] : java.lang.ThreadLocal$ThreadLocalMap$Entry[16]
    table : java.lang.ThreadLocal$ThreadLocalMap Size: 1
    threadLocals : java.lang.Thread Thread Name: pool-1-thread-3
  ]
  root = {
    description = N/A
    system = "Thread OopStorage"
    type = "Global Object Handle"
  }
  stackTrace = [
    AuditLeak$AuditEvent.<init>(String, String) line: 6
    AuditLeak.handleRequest(int) line: 15
    AuditLeak.lambda$main$0(int) line: 29
  ]
}

Read the object chain from the bottom up, and it is the whole diagnosis:

1. The root is a thread, pool-1-thread-3.
2. Its threadLocals field holds a ThreadLocalMap.
3. An entry in that map has a value: an ArrayList of 10,370 elements.
4. Element 9,625 is an AuditEvent, whose snapshot is the 1 KB array that was sampled.

And the stackTrace says where it was allocated: handleRequest, line 15. From "heap is growing" to "a thread-local list on a pool thread, filled in handleRequest" in one event, from a recording that costs a few percent of CPU.

#Classic leaks in services

The path to the root is usually one of these:

The fix follows the path: bound the map with a real cache (Caffeine with a maximum size), remove() the ThreadLocal in a finally, unregister the listener in a lifecycle hook, bound the queue.

static void handleRequest(int n) {
    try {
        PENDING.get().add(new AuditEvent("order-" + n, "PRICE_CHECKED"));
        flush();
    } finally {
        PENDING.remove();
    }
}

#Taking dumps safely in production

• Prefer JFR first. -XX:StartFlightRecording can run permanently in production at low overhead; the old-object sample is already there when a leak appears.
• A heap dump pauses the JVM for as long as it takes to write — seconds to minutes for a large heap. Take it from an instance that has been removed from the load balancer, or accept the pause deliberately.
• Write it to a disk with room: the file is about the size of the live heap.
• -XX:+HeapDumpOnOutOfMemoryError on every production JVM, so the dump exists at the moment the evidence is best.
• Treat the file as sensitive, restrict access, and delete it when the analysis is done.

#Verifying the fix

Run the fixed code under the same load for longer than the leak took to become visible, and compare the same measurement: used heap after GC, sampled over time. For this leak, the line from 21 MB to 90 MB should become flat. A leak "fixed" without that measurement is a leak you have stopped looking at.