Code10x
ReliabilityAdvanced

A logging and metrics platform that cannot take production down with it

The platform you use to debug an incident receives its heaviest load during that incident. Designing it means deciding, in advance, what it drops.

The brief

Every service ships logs and metrics to a central platform where engineers search logs, graph metrics and alert on them.

The platform serves a few hundred services, and its traffic rises sharply whenever something is going wrong.

Requirements

Functional

  • Collect logs from every service and search them by time, service and fields
  • Collect metrics and graph them over time
  • Alert on metric thresholds and rates
  • Keep recent data fast and older data cheap

Non-functional

  • A slow or unavailable platform must never slow or stop the services that send to it
  • Metrics and alerting must keep working when log ingestion is overloaded
  • Search over the last hour returns in seconds during an incident

Back-of-envelope

Assume

  • 300 services, 3,000 instances
  • Each instance logs 100 lines a second on a normal day, 1,000 during an incident; 500 bytes a line
  • Each instance exposes 1,000 time series, scraped every 15 seconds
  • Logs kept searchable 7 days, archived 90; metrics kept 13 months

Therefore

  • Normal log ingest: 3,000 × 100 × 500 B = 150 MB/s ≈ 13 TB a day raw. Seven days searchable, with indexing overhead and replication, is on the order of 150–200 TB of hot storage.
  • During an incident, error logging across affected services can rise tenfold: 1.5 GB/s. The platform's peak load and the moment you most need it are the same moment.
  • Metrics: 3,000 × 1,000 = 3 million active series, 200,000 samples a second. Metrics volume is set by the number of series, not by traffic — it stays flat during the incident that multiplies logs.
  • One user-ID label on one latency histogram with 10 buckets, in one service with 100,000 users: a million new series from a single line of code — a third of the platform's entire metric footprint.

The second and third lines are why logs and metrics are separate pipelines. The fourth is why the metrics pipeline needs its own guard, since its failure mode is a code change rather than a traffic change.

The interface

Agent → collector: batched, compressed log records over a persistent connectionServices write to local stdout or a file; a separate agent on each host ships it. The application's logging call never makes a network request, which is what keeps the platform out of the request path.
GET /metrics on every instance, scraped every 15 sPull, so the platform controls the rate it ingests. A service that suddenly produces more samples cannot push them faster than they are collected.
Query: logs { service, level, time range, field filters } · metrics { expression, range, step }Every log query is bounded by a time range, enforced by the API. An unbounded search across seven days is the query that saturates the cluster during an incident.

What is stored

log index (hot)one index per day · fields: timestamp · service · instance · level · trace_id · message · selected structured fields
Daily indexes make retention a matter of dropping a whole index rather than deleting documents. Only fields people filter on are indexed; the rest are stored but not indexed, which is most of the size saving.
log archive (cold)compressed files in object storage by service/date/hour
Ninety days of logs nobody searches daily. Restoring an hour into the hot tier for an investigation is slow and rare, and priced accordingly.
time-series databaseseries = metric name + label set → (timestamp, value) samples, downsampled after 15 days
Each distinct label combination is a series with its own storage and index entry, which is why label cardinality — not sample rate — decides the cost.

The design

Host agentsTail log files, batch, compress and send. They buffer to local disk when the collector is unreachable, up to a cap, and then drop the oldest data rather than fill the host's disk.
A durable bufferA log such as Kafka between collectors and indexers. Ingest keeps accepting while indexing is slow; lag grows instead of agents backing up.
Indexers and search clusterConsume from the buffer and write the hot indexes. Their throughput decides how far behind 'now' search results are during a spike.
Metrics and alertingA separate pipeline and a separate store, so that log ingestion falling hours behind does not delay a single alert.

The decisions

Each of these could go the other way. The choice, the reason, and what it costs — a design that lists only what it chose teaches the choice; one that lists what it gave up teaches the judgement.

Block the application when the log buffer is full, or drop?Chose — Drop, starting with the least important levelsBecause — A logging call that blocks turns a slow platform into slow requests in every service. An in-process async appender with a bounded queue that discards DEBUG and INFO first, when nearly full, keeps the service responsive.During the worst incidents some logs are lost, and they may be the ones that would have explained it. The drop count must itself be a metric, so a gap in logs is visible as a gap rather than silence.
One pipeline for logs and metrics, or two?Chose — TwoBecause — Log volume spikes with incidents while metric volume stays flat. Shared, a log flood delays metrics, and metrics are what alerting depends on — the tool that tells you about the incident would go quiet during it.Two systems to run, two query languages for engineers to learn, and correlation between a graph and its logs by trace id and timestamp rather than in one place.
Index every field of every log line?Chose — Index a chosen set; store the rest unindexedBecause — Full indexing can make the index larger than the raw logs. Most searches filter on service, level, time and trace id; the rest are read after the filter has narrowed the set.A search on an unindexed field is a slow scan, and someone will need it during an incident. Adding a field to the index applies only to data written from then on.

What breaks first

In order. Each names what you would actually observe, and each fix carries its cost.

Indexing during an incidentSymptom — Buffer consumer lag rises from seconds to tens of minutes exactly when error rates rise; engineers search for the last ten minutes and find nothing yet.Fix — Autoscale indexers on consumer lag, and prioritise ERROR and WARN topics over INFO so the useful lines index first.Indexer capacity that sits idle on normal days, and INFO logs that arrive an hour late during the incidents they might have explained.
Metric cardinalitySymptom — Memory on the time-series database climbs after a deploy with no traffic change; query latency degrades for every team, not just the one that deployed.Fix — A per-service series limit enforced at scrape time, rejecting samples beyond it and alerting the owning team.A team that legitimately needs more series is blocked until the limit is raised, and the rejected samples are simply missing.
Expensive queriesSymptom — The search cluster's CPU is saturated by a handful of broad queries over days of data, and everyone else's searches time out.Fix — Mandatory time bounds, per-user query concurrency limits, and timeouts on the query itself.An investigation that genuinely needs a week of logs must go through the archive, which is slower, at the moment someone is in a hurry.

When something fails

The log platform is completely downServices are unaffected: they write locally and agents buffer to disk up to their cap. Metrics and alerts continue on their own pipeline. When the platform returns, buffered logs arrive — out of time order, and the oldest may be gone if the outage outlasted the cap.
A service starts logging 50 times its usual volume in a loopWithout per-service ingest quotas it consumes the shared buffer and delays everyone's logs. With them, that service's excess is dropped at the agent or collector and its own drop metric rises — which is itself a useful alert.
The time-series database loses a nodeWith replicated ingestion, graphs and alerts continue from the replica. Without it, alerts evaluated against missing data must be configured to fire on absence, or an outage of the monitoring looks exactly like a healthy system.

Scaling it

Each step is triggered by a number, not a feeling — and carries what it costs.

10× servicesMove — More collectors, more buffer partitions and more indexers — the pipeline scales per stage — with quotas per service.Hot storage grows ten times, which is usually the point where retention and sampling are revisited rather than the cluster simply enlarged.
Log volume that storage budgets cannot followMove — Sample high-volume INFO logs at the agent, keep all ERRORs, and rely on traces for request-level detail.An individual request's INFO lines may not exist, so debugging a single customer's problem depends on the trace id being logged at the levels that are kept.
10× metric seriesMove — Shard the time-series database by metric or tenant and use recording rules to precompute common aggregations.Queries across shards are slower, and precomputed aggregations fix today's dashboards, not tomorrow's question.

What gets probed

The design is the easy half. These are where the conversation goes, and each has a defensible answer above.

  • The logging platform becomes slow. Show why no service's request latency changes.
  • An incident multiplies log volume by ten. What is dropped first, and how would an engineer know something was dropped?
  • Why is a user ID an acceptable log field and a dangerous metric label?
  • The monitoring system itself fails. What do your alerts do?