Make a service you can debug at 3am
Not a dashboard. The ability to answer 'which request was slow, and where did the time go' about a request that finished twenty minutes ago.
The business problem
Take a service you have already built and make it observable: structured logs with a correlation id, metrics that mean something, and traces that cross a service boundary.
The test is not that Grafana renders. It is that you can pick one slow request out of a million and say which call inside it was slow.
What you will have at the end
- Every log line carries a trace id
- Metrics that answer a question someone will actually ask
- A trace that spans two services
- An alert that fires on symptoms rather than on causes
Milestones
Each one ends in something you can observe. Without that a milestone is a heading, and you have no way to know you finished.
Structured logs
JSON, not a formatted string. A grep-able log is a log you cannot aggregate.
done whenYou can filter logs by field, not by regex.
Correlation
A trace id generated at the edge, propagated through every call, present on every line.
done whenOne id retrieves every log line for one request across both services.
Metrics worth having
Rate, errors, duration — per endpoint. Then the ones specific to you: pool usage, queue depth, consumer lag.
done whenYou can answer 'is it slow for everyone or for one endpoint' without adding code.
Distributed tracing
OpenTelemetry across a service boundary, with the database call as its own span.
done whenA trace shows the gateway, the service and the query, with the time attributed to each.
An alert you would not mute
Alert on p95 latency and error rate — symptoms. Alerting on CPU pages you for something that is often fine.
done whenYou can state, for each alert, what a human is supposed to do when it fires.