Actuator and observability

A service that cannot say whether it is healthy gets restarted on a guess. Actuator is Boot's answer to "how is this process doing": health that a platform can act on, metrics a dashboard can graph, and enough information about the running configuration to debug a deploy. It is also the part of a Boot application most often left exposed to the internet by accident, so this lesson spends as long on locking it down as on switching it on.

#What you get, and what is exposed

Add spring-boot-starter-actuator. Endpoints exist for health, info, metrics, env, configprops, beans, conditions, loggers, threaddump, heapdump, mappings and more, but by default only health is exposed over HTTP, at /actuator/health, and it answers with nothing but {"status":"UP"}. Everything else is opt-in:

management:
  endpoints.web.exposure.include: health,info,metrics,prometheus
  endpoint.health.show-details: when-authorized

Exposure is deliberately separate from enablement. An endpoint can exist, be exposed over JMX, and still not be reachable over the web. include: "*" is fine on a laptop and a finding in an audit.

#Health indicators and groups

Every auto-configured resource contributes a HealthIndicator: the datasource runs a validation query, Redis pings, Kafka checks the cluster, disk space checks a threshold. Overall status is the worst of them, DOWN maps to HTTP 503, and show-details decides whether the response says which one is down. Your own goes in as a bean:

@Component
class PaymentsGatewayHealth implements HealthIndicator {
    private final PaymentsClient client;
    PaymentsGatewayHealth(PaymentsClient client) { this.client = client; }
 
    @Override public Health health() {
        try {
            client.ping();
            return Health.up().build();
        } catch (Exception e) {
            return Health.down(e).withDetail("gateway", "unreachable").build();
        }
    }
}

A single /health that includes every dependency is the wrong shape for a platform to act on, and groups exist to fix that. A group is a named subset of indicators with its own URL:

management.endpoint.health.group:
  readiness.include: readinessState,db
  liveness.include: livenessState

#Liveness versus readiness

The two questions a platform asks are different, and answering both with the same indicator is how a database blip becomes a fleet-wide restart storm.

A database outage makes the service not ready; it does not make it dead. Restarting it will not bring the database back, and a hundred pods restarting at once and reconnecting is a thundering herd on top of an outage. So liveness includes only Boot's own livenessState, which goes BROKEN when the context fails, and readiness includes the dependencies. On Kubernetes, Boot detects the platform and exposes /actuator/health/liveness and /actuator/health/readiness automatically; elsewhere set management.endpoint.health.probes.enabled=true.

Readiness also flips by itself: ACCEPTING_TRAFFIC once the context is fully started, REFUSING_TRAFFIC at the beginning of graceful shutdown. Your code can publish it too, for a cache that must warm before the first request:

AvailabilityChangeEvent.publish(context, ReadinessState.REFUSING_TRAFFIC);

#Metrics through Micrometer

Boot instruments the things it owns and registers them with a MeterRegistry; micrometer-registry-prometheus on the classpath adds /actuator/prometheus in the text format a scraper wants. Out of the box: http.server.requests (tagged by uri, method, status, outcome, exception), JVM memory, GC pauses and threads, hikaricp.connections.*, Logback event counts, and with server.tomcat.mbeanregistry.enabled=true the Tomcat thread and connection gauges.

Your own metrics come from the same registry:

@Component
class OrderMetrics {
    private final Counter placed;
    private final Timer settlement;
    OrderMetrics(MeterRegistry registry) {
        placed = Counter.builder("orders.placed").tag("channel", "web").register(registry);
        settlement = Timer.builder("orders.settlement").publishPercentileHistogram().register(registry);
    }
    void recordPlaced() { placed.increment(); }
    <T> T timeSettlement(Supplier<T> work) { return settlement.record(work); }
}

The rule that keeps a metrics backend alive is bounded cardinality. A tag whose values are unbounded (user id, order id, a raw URL with an id in the path) creates a time series per value, and the backend falls over or the bill does. Boot already templates uri to /orders/{id} for this reason; keep your tags to enumerations.

Tracing is the third pillar: micrometer-tracing with the OpenTelemetry bridge propagates trace ids across RestClient, Kafka and JDBC and puts them in every log line as traceId. Set management.tracing.sampling.probability below 1.0 in production or the collector receives every request.

#Info and loggers

/actuator/info is where a deploy gets identified. With the build plugin's buildInfo() task and the git plugin, it returns the artifact version, the commit and the build time, so "which version is running in prod" is a curl, not a guess. /actuator/loggers is the runtime log-level switch: a POST to /actuator/loggers/com.acme.payments with {"configuredLevel":"DEBUG"} turns on debug for one package on one instance without a restart, and it is the endpoint that justifies putting Actuator behind authentication rather than firewalling it entirely.

#Securing endpoints

The exposure list is not security. env shows every environment variable name and, unless values are masked, the values; heapdump is the entire memory of the process including every session and secret in it; threaddump describes your code; loggers is writable. Layer the defences:

• Separate port. management.server.port=9090 moves Actuator off the application's connector, and the platform exposes 8080 to the load balancer and 9090 to nothing but the scraper and the probes. Most of the problem is solved before Spring Security is involved.
• Authenticate the rest. With Spring Security present, EndpointRequest.toAnyEndpoint() selects the Actuator paths; permit health (probes carry no credentials) and require a role for everything else.
• Keep values masked. env and configprops print ****** for every value in Boot 3 and later; management.endpoint.env.show-values=when-authorized is the most you should relax it.
• Leave shutdown disabled, which it is by default, and do not expose heapdump over the web at all; take heap dumps with jcmd on the box.

#Custom endpoints

When a service has operational state worth exposing, such as a circuit breaker to reset or a cache to inspect, write an endpoint rather than a controller so it inherits the exposure, port and security rules of the others:

@Component
@Endpoint(id = "cache")
class CacheEndpoint {
    private final CacheManager caches;
    CacheEndpoint(CacheManager caches) { this.caches = caches; }
 
    @ReadOperation Map<String, Integer> sizes() { /* name → entries */ }
    @DeleteOperation void clear(@Selector String name) { caches.getCache(name).clear(); }
}

GET /actuator/cache reads, DELETE /actuator/cache/{name} clears, and the endpoint appears in the discovery document with the rest.

#Under the hood: contributors, the aggregator, and what a scrape costs

/actuator/health is a HealthEndpoint over a HealthContributorRegistry holding every HealthIndicator (and CompositeHealthContributor) bean by name, with names derived from the bean name minus HealthIndicator. A request to the root endpoint calls each contributor, collects the Health values, and hands the statuses to a StatusAggregator, by default SimpleStatusAggregator, which orders them DOWN, OUT_OF_SERVICE, UP, UNKNOWN and returns the first present; HttpCodeStatusMapper then maps DOWN and OUT_OF_SERVICE to 503. A group is a filtered view over the same registry with its own aggregator and mapper, which is why a group can have different show-details and status rules. The indicators run sequentially on the calling thread by default, so the probe's latency is the sum of theirs, and an indicator with no timeout of its own inherits the probe's; management.endpoint.health.cache.time-to-live caches the whole result so a burst of probes does not multiply the work.

livenessState and readinessState are not checks at all: they read the ApplicationAvailability bean, an in-memory holder updated by AvailabilityChangeEvents. Boot publishes LivenessState.CORRECT once the context has refreshed and ReadinessState.ACCEPTING_TRAFFIC once ApplicationReadyEvent fires, and REFUSING_TRAFFIC when the context begins closing, which is what makes the readiness probe fail during graceful shutdown before Tomcat has finished draining. A liveness probe built only on this costs nothing, which is the point.

Metrics are Micrometer's MeterRegistry. http.server.requests is recorded by a WebMvcObservation filter around the DispatcherServlet, which also opens the tracing span; the uri tag is the matched handler pattern, so /orders/{id} is one series regardless of the id, and a request that matches no handler is tagged UNKNOWN, not with its raw path, precisely to keep cardinality bounded. Each meter is a name plus a sorted tag set; the Prometheus registry keeps them in a CollectorRegistry and renders the text format on every scrape by walking every meter, so a registry with 50,000 series costs a few hundred milliseconds per scrape and grows with each new tag value, forever, because Micrometer never expires a meter. That growth is what an unbounded tag does to a process: heap in the registry, CPU on every scrape, and a Prometheus server that hits its series limit.

#Walkthrough: the tag that took down the metrics backend

An orders service added a timer to track per-customer checkout latency.

void record(String customerId, Duration d) {
    Timer.builder("checkout.latency")
         .tag("customer", customerId)              // unbounded
         .register(registry).record(d);
}

1. Each new customer id created a new meter: name plus tag set. Micrometer's registry is a concurrent map keyed by that id, and a Timer with publishPercentileHistogram carries about seventy buckets. After a week, 400,000 customers meant 28 million series in the process's registry and a heap that grew 2 GB with no leak a profiler could name, because every object was legitimately referenced.
2. Every 15 s scrape walked all of it: 900 ms of CPU per scrape, on the request-serving JVM, and a 40 MB response. Prometheus hit its per-target series limit and dropped the target, so all metrics from the service vanished, including the ones the on-call dashboard used.
3. The alert that fired was "no data", which is the same alert as "the service is down", and the team spent the first twenty minutes on the wrong hypothesis.
4. The fix was to remove the tag: the timer became one series, and the per-customer question moved to where unbounded keys belong, a log line with the id and a trace. A MeterFilter bean, MeterFilter.maximumAllowableTags("checkout.latency", "customer", 100, MeterFilter.deny()), was added as a guard so the next unbounded tag would cap at a hundred values and drop the rest, and a test asserts that every tag key in the registry has fewer than a thousand values after a load run.
5. The registry's memory was only recovered by a restart. Micrometer does not expire meters, and that is the design: a series that disappears and reappears is worse for a time-series database than one that stays.

Metrics are for questions with a bounded set of answers. Anything keyed by an identifier is a log or a trace.

#Try it yourself

#Misconceptions

• "Health indicators run in the background." They run on the probe's request thread, sequentially, and the probe's timeout applies to their sum.
• "Liveness should check the database." It should check whether the process is broken. A dependency belongs in readiness, and only if a request cannot proceed without it.
• "Micrometer expires unused meters." It never does; every new tag value is a series for the life of the process. Bound tags to enumerations.
• "Exposing an endpoint on the management port is enough." The port limits the network path; env, heapdump and loggers still need authentication on that port, and heapdump should not be exposed at all.
• "/actuator/health returning UP means the service is fine." It means the worst of the included indicators was not DOWN, with details hidden. Groups say what was checked.

#Going deeper

• Spring Boot reference, "Production-ready Features": endpoints, health groups, Kubernetes probes, and "Securing HTTP Endpoints".
• HealthEndpointSupport, SimpleStatusAggregator and ApplicationAvailabilityBean in the Boot actuator source.
• Micrometer documentation, "Concepts → Naming meters" and "Meter filters"; the MeterFilter.maximumAllowableTags guard.
• Prometheus documentation, "Instrumentation → Labels", on cardinality, and promtool tsdb analyze to find the offending label.
• Spring Boot reference, "Observability" (Micrometer Observation and Tracing), for how a trace id reaches the log line.