Metrics with Micrometer and Prometheus

Logs tell you what happened to one request. Metrics tell you what is happening to all of them: how many, how fast, how many failed, how close to full. They are numbers aggregated over time, cheap enough to keep for months, and they are what dashboards and alerts are built on.

This lesson covers the tools — Micrometer in the service, Prometheus to collect — and the two ideas that decide whether the numbers mean anything: percentiles instead of averages, and cardinality.

#Micrometer, Prometheus, and pull

Micrometer is to metrics what SLF4J is to logging: a facade in your code, with a registry underneath for whichever monitoring system you use. Spring Boot Actuator configures it and records a large set of metrics without any code — HTTP server requests, JVM memory and GC, thread pools, connection pools, Kafka clients.

Prometheus collects them by pulling: every 15 or 30 seconds it requests /actuator/prometheus from each instance and stores the values as time series. Pulling means the monitoring system controls the rate, and an instance that stops answering is itself a signal.

The three meter types you will write yourself — reference code for the Micrometer API; the measurements later in this lesson come from a plain Java program, not a running registry:

// Counter: only goes up. Rate of change is what you graph.
Counter placed = Counter.builder("orders.placed")
        .tag("channel", "web")
        .register(registry);
placed.increment();
 
// Gauge: a value that goes up and down, read when scraped.
Gauge.builder("orders.pending", queue, Queue::size).register(registry);
 
// Timer: counts and measures durations, with a histogram when asked for.
Timer charge = Timer.builder("payment.charge")
        .publishPercentileHistogram()
        .register(registry);
charge.record(() -> gateway.charge(order));

A timer named payment.charge appears in Prometheus as payment_charge_seconds_count, _sum and, with the histogram enabled, _bucket — the names are converted to the monitoring system's conventions for you.

#What to measure: the golden signals

Starting from a blank dashboard is how teams end up with two hundred graphs nobody reads. Three well-known lists say what matters:

• The four golden signals (Google's SRE book): latency, traffic, errors, saturation.
• RED, for request-driven services: rate, errors, duration — the first three golden signals, per endpoint.
• USE, for resources like CPU, pools and queues: utilisation, saturation, errors.

For a typical Spring service that means requests per second, error rate and latency percentiles per endpoint; and for each pool — HTTP threads, database connections, executor queues — how full it is and how long work waits for it. Saturation is the one people leave out, and it is the one that moves before latency does, as the capacity planning lesson's latency curve shows.

#Averages lie

Here are 100,000 requests to one endpoint. Most take about 20 ms; 1.2% hit a slow path and take about two seconds. The distribution was generated with a fixed seed, and the statistics computed exactly:

one endpoint               mean   45.6   p50   20.1   p90   32.0   p99  1736.4   p99.9  2347.6   max  2399.5  (ms)

The mean is 45.6 ms. No request took 45.6 ms. Almost all took about 20, and about one in a hundred took nearly two seconds. The average blends a healthy majority with a broken minority into a number that describes neither, and a dashboard showing it looks fine.

The percentiles tell the truth: p50 is the typical request, p99 is the worst one in a hundred. For a page that makes ten backend calls, most page loads include at least one call from the slowest tenth, so the tail is what users actually feel.

#Percentiles do not add up

A service runs on two instances. Instance A handles 90% of the traffic and is healthy. Instance B handles 10%, and 5% of its requests take a slow path:

instance A                 mean   21.3   p50   20.0   p90   31.3   p99    45.2   p99.9    58.6   max    87.1  (ms)
instance B                 mean   91.5   p50   20.4   p90   35.5   p99  1675.5   p99.9  1787.9   max  1799.5  (ms)
average of the two p99s: 860.3 ms   true p99 of all requests: 49.2 ms

A dashboard that averages each instance's p99 shows 860 ms. The real p99 of the service's requests is 49 ms, because B's slow requests are only half a percent of all traffic — below the 1% the p99 looks at. Averaging percentiles gave a number seventeen times too large here, and with different traffic it can just as easily be too small.

Percentiles cannot be averaged, summed or combined. That is why Micrometer has two options that look similar and are not:

• publishPercentiles(0.99) computes the p99 inside each instance. Accurate for that instance, and impossible to combine correctly across instances.
• publishPercentileHistogram() publishes bucket counts. Counts can be summed across instances, and Prometheus computes percentiles from the sum:

histogram_quantile(0.99, sum by (le) (rate(http_server_requests_seconds_bucket[5m])))

Use histograms for anything you look at across more than one instance, which in production is everything.

#Histograms are estimates

A histogram stores how many requests fell into each bucket — under 5 ms, under 10, under 25 and so on — not the requests themselves. histogram_quantile finds the bucket containing the percentile and interpolates linearly inside it. The same 100,000 requests, with the exact value and two bucket layouts:

p50   exact    20.1   coarse buckets    20.1   buckets placed near the data    20.1
p95   exact    37.3   coarse buckets    46.8   buckets placed near the data    46.8
p99   exact  1736.4   coarse buckets  1278.5   buckets placed near the data  1584.6

The p95 is 37 ms, but it falls between the 25 ms and 50 ms boundaries, and interpolation says 47. The p99 falls between 1,000 and 2,500 in the coarse layout and comes out 26% low; adding boundaries near where the slow requests sit brought it within 9%. A histogram's accuracy is decided by where its buckets are, and the right place is near the values you care about.

That suggests the most useful change of all. Instead of asking "what is the p99?", ask "what share of requests finished within my target?" — and put a bucket boundary exactly on the target:

share of requests at or under 300 ms: 98.77%

With a boundary at 300 ms, that figure is a ratio of two counts — no interpolation, no estimate. In Spring Boot, management.metrics.distribution.slo.http.server.requests=300ms adds that boundary. It is exactly the number an SLO is written in, which the alerting lesson builds on.

#Cardinality: the metric that takes the platform down

Every distinct combination of tag values is a separate time series, with its own storage and its own entry in the index. The count multiplies:

http_server_requests  ×  30 endpoints  ×  4 methods  ×  6 status codes  ×  3 instances

That is up to 2,160 series for one metric, and multiplied by the buckets of a histogram — say 20 — about 43,000. Manageable. Now add a tag for the user id, with 100,000 users:

43,000 × 100,000 ≈ 4.3 billion possible series

No time-series database stores that. In practice the platform's memory climbs until it degrades for every team, and the cause is one line of code in one service. The logging and metrics platform design on this site names it as one of that platform's bottlenecks for that reason.

Spring's built-in HTTP metrics use the URI template for exactly this reason, and a request that matches no route gets a fixed value such as NOT_FOUND rather than its raw path — otherwise a scanner hitting random URLs would create a series per URL.