Code10x
Load balancingmedium3-5 years

An ECS service behind an ALB has its target group health check pointed at `/actuator/health`, which includes the database. During a 10-minute RDS incident the dashboard shows 0 healthy targets, yet some requests still reach the tasks, and ECS keeps replacing tasks that never finish starting. Explain the health check's semantics on an ALB, the fail-open behaviour, and what you would change.

An ALB health check does not restart anything: a target that fails it is removed from rotation, and returns when it passes again. So the check should be the readiness endpoint, and including the database there is correct in principle: no traffic to a task that cannot serve. Two AWS behaviours explain the rest. When every target in a group is unhealthy, the ALB fails open and routes to all of them anyway, which is why requests still arrived at '0 healthy'. And ECS adds a restart on top: a task whose target stays unhealthy is stopped and replaced, so a shared-dependency outage turns into churn, and new tasks that need longer than the health check grace period to start are killed before they are ready. Point the check at /actuator/health/readiness, set a grace period that covers a Spring Boot start, and read '0 healthy targets' during a database incident as the database, not the tasks.

The lesson behind it →
PreviousUsers upload invoices of up to 200 MB. The current endpoint accepts a multipart body in a Spring controller and streams it to S3 with the SDK; under load the service's heap and threads are exhausted. How would you redesign the upload, what does the service still own, and what does 'S3 is not a filesystem' mean for the rest of the design?Next Two failures in one week. (1) A service's role allows `s3:GetObject` on `arn:aws:s3:::reports/*`, nothing on the role denies anything, yet a batch job using a library that defaulted to plain HTTP gets `AccessDenied`. (2) An analytics account's role is given a bucket policy statement allowing it to read the same bucket, and it is still denied. Walk the evaluation for each.
More on Load balancing
Three requirements land at once for a public API that sits behind an ALB: a partner bank must allow-list a fixed set of IP addresses for your endpoint, the service's per-client rate limiter keys on the connecting IP and is currently limiting 'everyone at once', and a new gRPC API needs to be exposed. Explain the architectural difference between an ALB and an NLB that decides each requirement, and what you would build.