Audit Logging Platform
Audit logging ki requirements application logging se bilkul alag hain: IMMUTABLE (append-only, koi update/delete nahi), COMPLETE (ek bhi event miss nahi), aur TAMPER-EVIDENT (badla gaya to pata chale). Isliye ye "log ka ek aur type" nahi hai, alag system hai.
Tamper-evidence ke liye HASH CHAINING use hoti hai — har record mein pichhle record ka hash hota hai, isliye beech ka koi record badalne par poori chain toot jaati hai. Storage WORM (Write Once Read Many) hoti hai, jaise S3 Object Lock. Retention compliance-driven hoti hai (aksar 7 saal), isliye delete karna allowed hi nahi hota.
// Hash chain — beech ka record badla to chain toot jaayegi
record[n].hash = SHA256(
record[n].payload + record[n-1].hash
)
// Write path — kabhi drop nahi hona chahiye
Action → Audit Service → Kafka (durable, acked)
→ WORM storage (S3 Object Lock) + index (search ke liye)- Immutable, complete aur tamper-evident — normal logging se alag system
- Hash chaining se chhed-chhaad turant pakdi jaati hai
- WORM storage + compliance retention (delete allowed hi nahi)
Application log DEVELOPER ke liye hai — debugging, sampling allowed, delete allowed, best-effort delivery. Audit log COMPLIANCE ke liye hai — legal evidence, sampling kabhi nahi, delete kabhi nahi, guaranteed delivery.
Isliye inhe ek hi pipeline mein daalna galat hai. Audit events ko durable, acknowledged write chahiye — agar audit event store nahi ho paaya to ACTION HI FAIL hona chahiye. Application log ke liye ye behaviour bilkul galat hoga.
Auditors query karte hain — "is user ne pichhle 6 mahine mein kya kiya", "is record ko kis-kis ne dekha". Isliye actor, resource aur time par index chahiye. Par storage WORM hai, isliye search index alag rakha jaata hai (derived, rebuildable).
Retention compliance se aati hai — financial records 7 saal, healthcare 10 saal. Cost control ke liye purana data cold storage (Glacier) mein jaata hai jahan query slow par sasti hoti hai. Delete karna allowed hi nahi hota.
// WORM storage + alag search index
Audit event → Kafka (acked) → S3 Object Lock (immutable, 7 saal)
→ Elasticsearch (queryable, rebuildable)
// Index kho jaaye to S3 se rebuild ho sakta hai — S3 source of truth hai