Leader electionsenior8+ years

Design a distributed lock for a 5-node cluster where the lock protects writes to a shared storage service. A partition splits the cluster 2-3. Walk through, in order of strength, every defence that prevents both sides from believing they hold the lock and both actually writing.

Split brain is two nodes each believing it's the sole lock holder — a partition separates them, each side's detector concludes the other is dead, each acts alone, and if both write, the data diverges in a way no automatic merge can fix. The defences stack, each closing a gap the previous one leaves: first, a majority requirement — the lock is only granted by a quorum of the lock service's own nodes, so in a 2-3 split at most one side (the 3) can ever gather a majority, and the 2-side's lock requests simply fail, which is exactly why lock services run with three or five nodes and a two-node cluster can never be made safe this way. Second, even if something goes wrong with the majority check, fencing tokens at the storage service itself mean that only the newer token's writes actually land — a stale believer's write is refused by the resource, not merely discouraged. Third, stopping when unsure: a lock holder that fails to renew its lease stops acting as the holder immediately, rather than continuing until told otherwise, and a node that can't reach the quorum stops attempting writes at all. The combination — majority plus fencing plus stop-on-doubt — is what makes split brain structurally hard to reach, rather than something recovered from afterward.

The lesson behind it →
More on Leader election