Distributed Systems Coordination

Leader Election

Safely designating exactly one node as the leader in a distributed cluster — algorithms, fencing tokens, TTL-based leases, and split-brain prevention.

⏱ 9 min read

What it is

Leader election is the process by which a group of distributed nodes agrees on designating exactly one of themselves as the "leader" responsible for coordinating a shared task. The elected leader may perform writes to a shared resource, drive a distributed algorithm, schedule work, or manage cluster metadata. The rest of the nodes become followers and defer to the leader. When the leader fails, a new election is triggered to select a replacement.

Why it exists

Many coordination problems — replication, lock management, partition assignment, scheduled job execution — are simpler with a single authoritative node. Without a designated leader, every node must coordinate with every other node for every decision (O(n²) communication). Leader election concentrates decision-making, reducing coordination overhead to O(n) for follower-to-leader communication. The challenge is ensuring exactly one leader exists at any time, even across network partitions and process failures.

When to use

A single coordinator is needed for a distributed task: primary database writes, distributed job scheduling, partition leader in Kafka.
A service needs a primary replica to accept writes while secondaries replicate asynchronously.
Periodic tasks (distributed cron) must run on exactly one node to avoid duplicate execution.
You need a single authoritative cache invalidator or event processor.

When not to use

Leaderless designs (CRDTs, gossip-based eventual consistency) are sufficient and provide better availability.
The coordination overhead of election is not justified for the task.
You need to scale writes horizontally — a single leader creates a write bottleneck.

Typical architecture

ZooKeeper ephemeral nodes are a common leader election pattern:

ZooKeeper Leader Election via Ephemeral Sequential Nodes
═════════════════════════════════════════════════════════

All candidates create: /election/candidate-XXXXXXXX (sequential)

  Node A creates: /election/candidate-0000000001
  Node B creates: /election/candidate-0000000002
  Node C creates: /election/candidate-0000000003

Leader = node holding the lowest-numbered znode

  → Node A is leader (has candidate-0000000001)
  → Node B watches candidate-0000000001
  → Node C watches candidate-0000000002 (watches next lower)

Node A crashes → its ephemeral znode is deleted automatically
  → Node B's watch fires → Node B becomes leader
  → Node C watches candidate-0000000002 (now B's node)

Fencing token: leader is assigned epoch/term number.
Storage guards reject writes with outdated epoch.

In Raft, leader election is built into the consensus protocol: nodes use randomised election timeouts; the first to time out sends RequestVote RPCs; a candidate gaining majority votes becomes the leader for that term.

Pros and cons

Pros

Simplifies coordination — followers defer to a single authoritative node.
Reduces inter-node communication compared to fully decentralised coordination.
ZooKeeper and etcd provide battle-tested, off-the-shelf leader election primitives.
Fencing tokens provide safety even when the old leader has not yet realised it has been replaced.

Cons

Leader is a single point of failure for writes; follower promotion takes time (seconds to tens of seconds).
Split-brain risk: a slow or partitioned leader may believe it is still the leader and continue writing.
Extra infrastructure dependency if using ZooKeeper or etcd for election.
Bully algorithm is simple but generates O(n²) messages in large clusters.

Implementation notes

Fencing tokens are the key safety mechanism for preventing split-brain damage. Each time a new leader is elected, it receives a monotonically increasing token (term number, epoch, or ZooKeeper session ID). The leader includes this token in every write request to storage. The storage system rejects writes with tokens lower than the highest it has seen, preventing a zombie (old) leader from overwriting data after a new leader has been elected.

TTL-based leases with heartbeats: the leader holds a lease that expires after a TTL unless renewed via heartbeat. Followers will not act as leader until the lease expires. The leader must stop acting as leader before its lease expires if it loses connectivity. This approach works well with Redis (SET NX PX) or Kubernetes leader election via ConfigMap/Lease annotations. Ensure clocks are reasonably synchronised — clock drift can cause a lease to expire on the leader's clock before the TTL has elapsed on followers.

Epoch numbers serve as fencing tokens in Kafka, where each partition has a leader epoch written to ZooKeeper (pre-KRaft) or to the metadata log (KRaft mode). Consumers and other brokers reject writes from brokers with outdated epochs.

Common failure modes

Split-brain without fencing: network partition causes a new leader to be elected while the old leader is still running, both accepting writes.
False failure detection: a slow or GC-paused leader is incorrectly declared dead, triggering an unnecessary election and disrupting in-flight operations.
Thundering herd: all nodes detect leader failure simultaneously and spam election messages, delaying new leader selection.
TTL too short: frequent re-elections under load due to heartbeat timeouts caused by CPU saturation or GC pauses.
TTL too long: slow failover when the leader genuinely fails, causing extended unavailability.

Decision checklist

Are fencing tokens (epoch numbers) implemented to prevent split-brain writes?
Is the election timeout tuned relative to your network latency and GC pause budget?
Does your storage layer validate fencing tokens before accepting writes?
Is there a runbook for scenarios where no leader can be elected (quorum loss)?
Have you tested leader election under network partitions and GC pauses?
Is there monitoring for leader changes and election frequency?

Example use cases

Kafka partition leaders elected via ZooKeeper (or KRaft) — the leader handles all produce and fetch requests for that partition.
Kubernetes controller-manager and scheduler use leader election via Lease API to ensure only one instance runs active reconciliation loops.
Database primary election in PostgreSQL streaming replication managed by Patroni (using etcd or Consul).
Distributed cron job: multiple application instances compete; winner runs the scheduled task.

Consensus Algorithms — Raft embeds leader election as part of the consensus protocol.
Distributed Locking — leader election is implemented using a distributed lock that only one node holds.
Split Brain Problem — the central hazard that leader election and fencing tokens are designed to prevent.