p95 went from 180ms to 4.2 seconds
A Spring Boot service that has been stable for months degraded over about twenty minutes. It has not crashed and no deploy went out. These are the numbers on the dashboard.
There is nothing to run here. Say what you would look at first and why — the order is the answer, not the diagnosis.
Constraints
- No deploy went out. The change is in the environment or the data, not the code.
- The service is not throwing — whatever is happening, it is succeeding slowly.
Hints
Hint 1
CPU at 35% with threads at 92% means they are waiting, not working. Find what they wait on.
Hint 2
The pool is full and the database is at 20% CPU. The database is not the bottleneck; getting to it is.
Hint 3
A connection held for the length of an HTTP call is the classic version of this — look for a transaction spanning a remote call.
Stuck? The lesson behind this problem: 🚀 Actuator and observability
What the dashboard says
API p95180ms → 4.2s
CPU35%
Heap used62%
GC pauseunchanged
DB connection pool100% in use
Tomcat threads92% busy
DB CPU20%
Error rate0.02%, unchanged
Every one of these is a real thing that produces slow requests. Which one do these numbers actually support?
Pick one first — the answer stays hidden until you do.