Code10x
Observabilitysenior8+ years

You inherit an alarm set for a Spring Boot service behind an ALB and RDS: `TargetResponseTime` Average above 1 s, `HTTPCode_Target_5XX_Count` above 10, CPU above 80%. It pages at night for single errors, missed a 20-minute outage in which the tasks crashed and traffic stopped, and never caught a slow p99. As the owner of on-call quality, redesign it and explain the CloudWatch mechanics each change depends on.

Each failure maps to a CloudWatch mechanism. A metric is not a live reading; it is an aggregate per period (one minute by default for the ALB's metrics), read through a statistic you choose. Average hides a slow tail inside a fast minute, so latency alarms use a percentile (p99) against the SLO. A raw count of 5xx pages for one error on a quiet night and misses a real error rate at peak, so errors are a ratio (metric math over RequestCount) sustained for several periods. The outage was missed because a period with no requests publishes no data point: the alarm went to INSUFFICIENT_DATA, which is not ALARM, and nobody had set treat missing data. So: alarm on symptoms (error ratio, p99 latency, HealthyHostCount, the load balancer's own 5xx), choose missing-data handling per alarm, drop CPU as a pager, compose related alarms so an incident pages once, and put the runbook in each alarm's description.

The lesson behind it →
PreviousYour team runs RDS PostgreSQL Multi-AZ and treats it as 'automatic HA, nothing for the application to do'. During a failover drill, one service recovers in about a minute while another keeps failing queries for much longer, until its tasks are restarted. Explain what Multi-AZ replication and failover actually are, why the two services behaved differently, and what the application must configure.Next Three requirements land at once for a public API that sits behind an ALB: a partner bank must allow-list a fixed set of IP addresses for your endpoint, the service's per-client rate limiter keys on the connecting IP and is currently limiting 'everyone at once', and a new gRPC API needs to be exposed. Explain the architectural difference between an ALB and an NLB that decides each requirement, and what you would build.