Metrics Monitoring System
Metrics logs se BILKUL alag hain — ye numeric time series hain (timestamp, metric name, labels, value), isliye purpose-built TIME SERIES DATABASE (Prometheus, VictoriaMetrics) use hota hai jo delta aur compression se 10-20x kam jagah leta hai.
Do design decisions matter karte hain. PULL vs PUSH — Prometheus pull karta hai (service discovery se targets scrape), jo simple hai par short-lived jobs ke liye push gateway chahiye. Aur DOWNSAMPLING/ROLLUP — raw 10-second data 15 din rakho, 5-minute average 90 din, 1-hour average 2 saal. Bina rollup ke storage explode ho jaata hai.
// Cardinality bomb — sabse common production outage
http_requests{path="/users", status="200"} ✅ ~50 series
http_requests{path="/users", user_id="12345"} ❌ crore series
// Rollup — storage linear rehta hai
raw (10s) -> 15 din
5-min avg -> 90 din
1-hour avg -> 2 saal- Time series DB use karo — compression se 10-20x jagah bachti hai
- Pull (Prometheus) vs push ka trade-off batao
- Rollup zaroori; high cardinality labels sabse bada khatra
Alerts SYMPTOM par hone chahiye, CAUSE par nahi. "Error rate 5% se upar" achha alert hai (user affected hai). "CPU 80%" bura alert hai — ho sakta hai sab theek chal raha ho.
Alert fatigue asli problem hai — agar roz 50 alerts aayein to log unhe ignore karne lagte hain aur asli incident miss ho jaata hai. Isliye alerts par threshold + DURATION dono lagao ("5 minute tak error rate 5% se upar"), aur alerts ko severity mein baanto.
# Symptom-based, duration ke saath — flapping se bachta hai
- alert: HighErrorRate
expr: rate(http_errors[5m]) / rate(http_requests[5m]) > 0.05
for: 5m # 5 min tak sach ho tabhi alert
labels: { severity: page }PULL (Prometheus): monitoring system targets ko scrape karta hai. Fayde — target ki health apne aap pata chal jaati hai (scrape fail = down), configuration central hai, aur target ko monitoring system ka pata nahi hona chahiye.
PUSH (StatsD, OTLP): services metrics bhejti hain. Ye short-lived jobs (batch, serverless) ke liye zaroori hai jo scrape hone tak zinda hi nahi rehte. Practical systems dono support karte hain — pull default, push gateway short-lived kaam ke liye.