Distributed File Storage
Sabse important design decision hai METADATA aur DATA ko alag karna. File content blob storage mein chunks ke roop mein jaata hai; metadata (naam, path, permissions, chunk list) ek database mein. Dono ki scaling needs bilkul alag hain — metadata chhota par transactional, data bada par simple.
File ko fixed-size CHUNKS mein todo (jaise 4 MB). Isse teen fayde hain: parallel upload, resume-on-failure, aur DEDUPLICATION — har chunk ka hash nikaalo, wahi chunk pehle se ho to dobara store mat karo. Sync ke liye sirf badle hue chunks bhejo (delta sync), poori file nahi.
// Metadata (small, transactional) aur data (large, simple) alag
File metadata DB:
file_id, name, path, owner, version,
chunks: [hash1, hash2, hash3, ...]
Blob storage:
hash1 -> 4MB blob (replicated 3x across zones)
// Dedup: same hash = pehle se hai, dobara upload nahi
// Delta sync: sirf badle hue chunk hashes bhejo- Metadata (DB) aur data (blob storage) hamesha alag
- Fixed-size chunks — parallel upload, resume, aur dedup teeno milte hain
- Delta sync se sirf badle chunks jaate hain
Durability ke liye har chunk ki 3 copies alag FAILURE DOMAINS mein rakhi jaati hain (alag rack, alag availability zone). Ek zone jaane par data available rehta hai.
3x replication mehnga hai — 1 PB data ke liye 3 PB storage. Isliye cold data ke liye ERASURE CODING use hoti hai: data ko k chunks mein todo aur m parity chunks banao. 1.5x storage mein wahi durability milti hai, par reconstruction slow hota hai. Hot data replication par, cold data erasure coding par.
// Replication: simple, fast read, 3x cost
chunk → zone-a, zone-b, zone-c
// Erasure coding (6+3): 1.5x cost, slow rebuild
data → 6 data chunks + 3 parity chunks
→ koi bhi 6 chunks se poora data wapasDo devices offline mein same file badal dein to conflict hota hai. Silently ek ko overwrite karna sabse bura option hai — user ka kaam gum ho jaata hai.
Practical solution: dono versions rakho aur ek ko "conflicted copy" naam se dikhao (Dropbox yahi karta hai). Automatic merge sirf structured data ke liye possible hai (CRDTs), binary files ke liye nahi.