Alerting and SLOs

An alert is a decision to interrupt a person. Every alert that fires without a real problem teaches that person to ignore the next one, and every real problem that fires no alert is found by a customer instead. Most alerting fails in both directions at once, because it is built on thresholds that were chosen by feel.

Service level objectives replace the feel with a number the business agreed to, and burn-rate alerts turn that number into pages that are both earlier and rarer. This lesson builds both, and then measures them against a static threshold over a simulated month.

#SLI, SLO, SLA

• An SLI (service level indicator) is a measurement of something users experience, expressed as a ratio of good events to all events: the share of requests that succeeded, the share that completed within 300 ms.
• An SLO (objective) is the target for that ratio over a window: 99.9% of checkout requests succeed, measured over 30 days.
• An SLA (agreement) is a contract with consequences — refunds, credits — usually set looser than the SLO, so the team hears about trouble before the contract does.

A good SLI is measured as close to the user as possible and counts what the user would count. "The database is up" is not an SLI; "checkout requests that returned a success in under a second" is. The metrics lesson's histogram with a bucket boundary exactly on the latency target is what makes the latency SLI a count rather than an estimate.

#The error budget

An SLO of 99.9% means 0.1% of requests are allowed to fail. That allowance is the error budget:

30 days × 24 h × 60 min = 43,200 minutes
0.1% of 43,200            = 43.2 minutes of total outage a month
or, by requests:            1 failure in every 1,000

The budget changes the conversation. A 100% target makes every change a risk to be avoided; a budget makes risk something to spend. While budget remains, the team ships. When a month's budget is gone, the agreed error budget policy applies — typically, a freeze on risky changes and time spent on reliability until the budget recovers. The value of the policy is that it was agreed before anyone was arguing about a specific release.

#Alert on symptoms, not causes

A cause-based alert fires on something that might hurt users: CPU above 80%, a pod restarted, one replica lagging. A symptom-based alert fires on something that is hurting users: errors, latency, the SLI itself.

Cause alerts are both noisy and incomplete. High CPU during a batch job hurts nobody, and pages anyway. A misconfigured feature flag that fails every checkout uses no extra CPU, and pages nobody. Page on symptoms; keep cause-based signals on dashboards and in tickets, where they help diagnosis without waking anyone.

#Burn rate

The burn rate is how fast the error budget is being spent, relative to the rate that would spend exactly all of it by the end of the window:

burn rate = observed error rate / error rate the SLO allows
 
0.1%  errors against a 99.9% SLO  → burn rate 1     (budget lasts exactly 30 days)
1.44% errors                      → burn rate 14.4  (a month's budget in about 2 days)
100%  errors                      → burn rate 1,000 (a month's budget in 43 minutes)

An alert on burn rate asks the question that matters — at this pace, will we miss the objective? — instead of "is the error rate above a number somebody picked?".

The pattern described in the Google SRE Workbook uses several windows at once:

The long window makes the alert significant: it fires only when enough budget has been spent to matter. The short window makes it recover quickly: once the problem stops, the short window's rate falls and the alert resolves, instead of staying red for the rest of the hour.

#A month, measured

A simulated service, one minute at a time for 30 days, with an SLO of 99.9%. Traffic follows the day — about 40 requests a minute at 4am, about 2,000 in the evening — and there is a little background failure, plus a client that about once a day retries one bad request six times. Three incidents happen:

• A — a bad deploy: every request fails for 4 minutes.
• B — a slow burn: 0.8% of requests fail for 36 hours.
• C — a partial outage: 20% of requests fail for 45 minutes.

The random numbers are seeded, so the run repeats exactly:

30 days: 37,052,353 requests, 43,743 errors, availability 99.8819%, error budget used 118%
  incident A used   11% of the month's budget
  incident B used   49% of the month's budget
  incident C used   29% of the month's budget

The month missed its objective. The incident that did most of the damage was B, the quietest: under 1% errors, no graph that would look alarming at a glance, and half the month's budget.

Two alerting rules watched the same traffic:

static: error rate > 1% over 5 minutes
  pages 20, of which with no incident: 11
  incident A: first page after 0 min, 1 page(s)
  incident B: first page after 15 min, 7 page(s)
  incident C: first page after 0 min, 1 page(s)
burn rate: 14.4× over 1h and 5m, or 6× over 6h and 30m
  pages 3, of which with no incident: 0
  incident A: first page after 1 min, 1 page(s)
  incident B: first page after 209 min, 1 page(s)
  incident C: first page after 4 min, 1 page(s)

(A page is counted when a rule starts firing after at least an hour of silence, so one flapping alert is not counted as dozens.)

The static threshold paged 20 times, and 11 of those were nothing. Every one of them was at night, when forty requests a minute means six errors from one misbehaving client is "more than 1% over five minutes". Nobody could have fixed anything. Eleven wake-ups a month for nothing is how an on-call rotation learns to acknowledge and go back to sleep.

It also paged seven times for incident B. The slow burn ran just under the 1% line, so random variation pushed it over and back repeatedly. Seven separate pages for one problem, each resolving itself minutes later, look like flapping noise rather than an incident — which is exactly how a real slow burn gets ignored.

The burn-rate rules paged three times, once per incident, and never for nothing. At night, six errors in an hour of 2,400 requests is a burn rate of about 2.5 — nowhere near a threshold. A and C paged within minutes.

The honest cost is on B: 209 minutes before the six-hour window accumulated enough evidence. At a burn rate of 8, those three and a half hours spent roughly 4% of the month's budget before anyone was paged. That is the design working as intended — a problem that spends the budget slowly is allowed to be found slowly — and the three-day ticket rule exists to catch burns too slow even for that.

#Getting started without all of it

A service with no SLOs does not need the full table on day one:

1. Pick one or two SLIs per user-facing flow: success ratio, and share of requests under a latency target.
2. Set an SLO users would actually notice being missed, and write down the error budget in minutes.
3. Add one burn-rate page — 14.4× over an hour with a five-minute short window — and one burn-rate ticket.
4. Delete, or demote to tickets, the cause-based pages that the SLO alert makes redundant.

The rest — more windows, per-customer SLOs, budget policies — is worth adding once the first alert has proved itself.