API Analytics Platform

High Volume Ingestion Aur Aggregation

💡 API analytics TOLL PLAZA KA HISAAB hai. Har gaadi ka record rakhna hai, par report mein kisi ko har gaadi nahi chahiye — per-ghanta, per-route summary chahiye.

Ye classic ingestion + aggregation problem hai. Raw events (endpoint, status, latency, client) bahut zyada hote hain, isliye raw ko lambe samay tak rakhna mehnga hai. Lambda ya Kappa architecture use hota hai: stream processing (Flink/Kafka Streams) se real-time aggregates, aur batch se accurate historical rollups.

Queries hamesha AGGREGATE hoti hain — "p99 latency per endpoint per hour", "error rate by client". Isliye pre-aggregated rollups columnar store (ClickHouse, Druid) mein rakho. Percentiles ke liye exact computation mehnga hai, isliye approximate sketches (t-digest, HdrHistogram) use hote hain jo mergeable hote hain.

// Ingestion → real-time aggregation → columnar store
API Gateway → Kafka (raw events)
            → Flink (1-min windows: count, error rate, latency sketch)
            → ClickHouse (pre-aggregated rollups)

// Percentile sketches mergeable hote hain —
// 60 one-minute sketches jodkar hourly p99 mil jaata hai

📊

API analytics TOLL PLAZA KA HISAAB hai. Har gaadi ka record rakhna hai, par report mein kisi ko har gaadi nahi chahiye — per-ghanta, per-route summary chahiye.

1 / 2

⚡ Quick Recap

Raw events Kafka → stream aggregation → columnar store
Pre-aggregated rollups par query karo, raw par nahi
Percentiles ke liye mergeable sketches — averages galat jawab dete hain

Is page mein (2 subtopics)

LAMBDA architecture mein do paths hote hain — speed layer (real-time, approximate) aur batch layer (slow, accurate). Results merge hote hain. Problem: same logic do jagah likhni aur maintain karni padti hai.

KAPPA architecture mein sirf stream processing hoti hai; historical data ko dobara process karna ho to stream REPLAY kar do. Ye simpler hai aur aaj zyada popular hai, kyunki Kafka lambe retention aur replay dono deta hai.

💡Tip: Aaj ke interviews mein Kappa ko default bolo aur reason do — "ek hi code path, replay se reprocessing." Lambda ka naam jaanna achha hai par usse shuru mat karo.

Aggregation dimensions par hoti hai (endpoint, status, client, region). Har extra dimension combinations ko multiply karta hai — 100 endpoints × 10 statuses × 1000 clients = 10 lakh combinations per time window.

Isliye pre-aggregation ke liye dimensions carefully choose karo. High-cardinality fields (user_id, request_id) ko aggregate dimension mat banao — unhe raw events mein rakho aur zaroorat par hi query karo.

// ✅ Pre-aggregate — bounded cardinality
(endpoint, status, region, minute) -> count, p50, p99

// ❌ user_id dimension mein daalna — cardinality explode
(endpoint, status, user_id, minute)   // crore rows/minute

← Previous🔏 Audit Logging Platform Next →🗄️ Distributed File Storage

Code10x

API Analytics Platform

Lambda vs Kappa Architecture

Cardinality Aur Pre-Aggregation