Distributed Job Scheduler
Do parts hain: SCHEDULING (kab chalana hai) aur EXECUTION (kahan chalana hai). Due jobs ek durable store mein rehti hain (time par indexed). Workers atomically job CLAIM karte hain — conditional update se ("SET status=RUNNING WHERE status=PENDING AND id=?"), jo ek hi worker ko safal karta hai.
Exactly-once ka sach batana zaroori hai: distributed systems mein exactly-once delivery possible NAHI hai. Jo milta hai wo at-least-once delivery plus IDEMPOTENT jobs hai — matlab job do baar chalne par bhi result same rahe. Ye batana theoretical maturity dikhata hai. Worker crash hone par lease/heartbeat timeout se job wapas queue mein aa jaati hai.
-- Atomic claim — sirf ek worker jeetega
UPDATE jobs
SET status = 'RUNNING', worker_id = :me, lease_until = now() + 60s
WHERE job_id = :id AND status = 'PENDING';
-- affected rows 1 => tumne claim kiya, 0 => koi aur le gaya
-- Worker crash: lease_until nikal gaya => job wapas PENDING- Atomic conditional claim se do workers ek job nahi utha sakte
- Lease + heartbeat — worker crash par job wapas available
- Exactly-once delivery impossible hai; at-least-once + idempotency use karo
Har second "kaunsi jobs due hain" query karna DB par bhaari padta hai. Solution: time par index rakho aur bounded window query karo ("agle 10 second mein due").
Bade scale par jobs ko TIME BUCKETS mein baanto — har minute ka apna bucket. Scheduler sirf current aur agla bucket dekhta hai, poora table nahi. Ye query ko constant-time bana deta hai chaahe crore jobs pending hon.
-- Bounded window, indexed
SELECT * FROM jobs
WHERE status = 'PENDING'
AND run_at BETWEEN now() AND now() + interval '10 seconds'
LIMIT 1000;
-- Time buckets: jobs:2026-08-03T14:32 -> us minute ki jobsAgar scheduling ka faisla ek jagah hona chahiye (jaise cron expressions evaluate karna) to leader election chahiye — Zookeeper, etcd ya Redis lock se. Ek leader decide karta hai, baaki standby rehte hain.
Par EXECUTION ke liye leader nahi chahiye — workers stateless hain aur atomic claim se coordinate kar lete hain. Ye distinction important hai: leader election ko har jagah lagana bewajah complexity aur single point of failure hai.