Centralized Logging System
Pipeline fixed hai: AGENT har machine par (Fluent Bit, Filebeat) → BUFFER (Kafka) → PROCESSING (parse, enrich) → INDEX (Elasticsearch) → QUERY UI (Kibana). Kafka beech mein isliye hai ki Elasticsearch down ho ya slow ho to logs khoyein nahi — buffer absorb kar leta hai.
Cost hi asli constraint hai. Terabytes roz index karna bahut mehnga hai, isliye TIERED RETENTION use hoti hai: 7 din hot (fast SSD, fully indexed), 30 din warm, 1 saal cold (S3, sirf search-on-demand). Aur SAMPLING — high-volume DEBUG logs ka sirf 1% rakho, ERROR ka 100%.
Agent (Fluent Bit) → Kafka → Processor → Elasticsearch (hot 7d)
↓ rollover
Warm (30d) → S3 (1y)
// Sampling: volume 90% ghata deta hai
ERROR/WARN -> 100% rakho
INFO -> 10%
DEBUG -> 1%- Agent → Kafka buffer → process → index → query, ye fixed pipeline hai
- Kafka backpressure absorb karta hai; index down hone par log nahi khota
- Tiered retention + sampling — cost hi asli constraint hai
Agent har machine par chalta hai aur uski sabse badi zimmedaari hai APPLICATION KO NA GIRANA. Agent local disk par buffer karta hai, batches mein bhejta hai, aur pipeline slow hone par apne buffer se kaam chalata hai.
Buffer bhar jaaye to policy chahiye — purane logs drop karo (application chalti rahe) ya likhna rok do. Logging ke liye lagbhag hamesha DROP sahi jawab hai: log khona acceptable hai, production service girana nahi.
Elasticsearch mein TIME-BASED INDICES banaye jaate hain (logs-2026-08-03). Isse purana data delete karna ek index drop hai — document-by-document delete ke muqable hazaar guna tez.
Har field ko index mat karo — jo fields kabhi search nahi hote unhe store karo par index nahi. Ye index size aur ingestion cost dono bahut ghata deta hai, aur bade deployments mein ye sabse bada cost lever hai.
// Time-based indices — retention ek drop operation ban jaata hai
logs-2026-08-01, logs-2026-08-02, logs-2026-08-03 ...
DELETE logs-2026-07-* // instant, 7 din se purana gaya
// Field mapping: sab kuch index mat karo
{ "message": { "type": "text", "index": true },
"raw_payload": { "type": "text", "index": false } }