Distributed Coordination

What it is

Distributed coordination services provide strongly-consistent, highly-available primitives — key-value storage, watches (event notifications), leases (TTL-bound ownership), and atomic compare-and-swap — that distributed applications use to build higher-level coordination protocols. ZooKeeper, etcd, and Consul are the dominant implementations. They are intentionally narrow in scope: they prioritise consistency (CP in CAP) and expose simple primitives rather than rich data models, trusting applications to compose them into election, locking, and configuration systems.

Why it exists

Building correct distributed coordination from scratch (leader election, distributed locking, cluster membership) requires implementing consensus algorithms correctly — a notoriously hard problem with subtle failure modes (see: split-brain, fencing tokens). Dedicated coordination services encapsulate this complexity behind a simple, well-tested API. Rather than every team implementing their own Paxos or Raft variant, they delegate coordination concerns to proven systems, focusing on their application logic.

When to use

• Leader election for a single-primary database, scheduler, or worker process.
• Distributed locks protecting shared resources that cannot be protected by database transactions.
• Service registry for dynamic membership — tracking which instances are alive.
• Distributed configuration management — cluster-wide config that services watch for changes.

When not to use

• Not a general-purpose database — data must be small (etcd recommends <8 GB total, key-value pairs <1.5 MB).
• Not suitable for high-throughput data storage — throughput is bounded by consensus latency (hundreds to low thousands of writes/sec).
• Do not use as a message queue, cache, or blob store.

Typical architecture

ZooKeeper Node Types
═════════════════════
  /config              ← persistent node (configuration)
  /config/db-url       ← persistent node (value)
  /election            ← persistent node (parent)
  /election/n-0000001  ← ephemeral sequential (candidate 1)
  /election/n-0000002  ← ephemeral sequential (candidate 2)

  Watch mechanism:
    Client watches /election → notified when child created/deleted
    Lowest sequential node = current leader
    On leader crash: ephemeral node deleted → watchers notified

etcd Key Operations
════════════════════
  PUT /config/db-url "postgres://..."
  GET /config/db-url
  WATCH /config/           ← streams all changes under prefix
  PUT /lock/my-service --lease=  ← TTL-bound ownership
  TXN [IF key == ""] THEN [PUT key val] ← atomic CAS

Consul Features
════════════════
  Service registration with health checks
  Key-Value store (similar to etcd)
  Sessions (TTL-based leases for locking)
  DNS interface for service discovery
  Connect (service mesh with mTLS via Envoy)

Pros and cons

Implementation notes

ZooKeeper exposes a hierarchical znode filesystem. Znodes can be persistent (survive client disconnection), ephemeral (deleted when the creating client session ends), or sequential (node name gets a monotonically increasing suffix). These primitives compose into recipes: leader election (lowest sequential ephemeral node wins), group membership (ephemeral children of a parent node), distributed queues, and barriers. ZooKeeper's ZAB protocol provides total order broadcast with linearisable reads on quorum nodes.

etcd provides a flat key-value API with prefix-based ranging and watches. Its MVCC (Multi-Version Concurrency Control) model retains a compactable history of all writes, enabling snapshot-based watches. Leases (TTL-bound tokens) allow any key to be associated with a lease; if the lease is not renewed before TTL expiry, all associated keys are atomically deleted. Transactions (TXN) provide conditional multi-key atomic operations. etcd's gRPC API is cleaner than ZooKeeper's bespoke protocol, and it is the standard coordination store for Kubernetes.

Performance characteristics: A 3-node etcd cluster on commodity hardware typically handles 2,000–5,000 write ops/sec, with read throughput much higher when using linearisable reads from the leader. ZooKeeper performs similarly. Both are far below what a standalone database or cache can handle — they are not intended for data-plane operations. Configuration and coordination data is typically small and infrequently changed, so these limits are rarely binding.

Common failure modes

• Watch event loss during reconnect: ZooKeeper and etcd watches are one-shot; clients must re-establish watches and re-read state after reconnection to avoid missing events during the gap.
• etcd disk I/O saturation: etcd writes all entries to a WAL before acknowledging. On systems with high write latency (e.g., NFS or overloaded disks), etcd latency spikes and may lose leadership. etcd requires fast local SSDs.
• Session expiry and lock release: if a lock holder's session expires (network partition, GC pause), the lock is released. The holder must detect this and stop performing the critical section — fencing tokens address this.
• Unbounded key growth: systems that write unique keys without compaction fill etcd's keyspace, degrading performance and eventually hitting the storage limit.

Decision checklist

• Are you using Kubernetes? etcd is already present — leverage it via Kubernetes primitives (Leases, ConfigMaps) rather than running a separate coordination cluster.
• Do you need service discovery in addition to coordination? Consul provides both with a unified API.
• How large is your coordination data? If >1 MB per value, a coordination service is the wrong tool.
• Have you tested watch reconnection logic and session expiry behaviour under network partition?
• Is the coordination cluster on local fast SSDs? Network-attached storage is unsupported for etcd WAL.

Example use cases

• Kubernetes using etcd for all cluster state (Pod definitions, deployments, service configurations) with Raft consensus across etcd nodes.
• Apache Kafka using ZooKeeper (replaced by KRaft in Kafka 3.x) for broker leader election and partition assignment coordination.
• HBase using ZooKeeper for region server master election and distributed coordination of region assignments.
• Consul used by HashiCorp Nomad for job scheduler leader election, cluster membership, and service registration.

Related patterns

• Consensus Algorithms — ZooKeeper uses ZAB; etcd and Consul use Raft; understanding consensus is prerequisite to reasoning about coordination service guarantees.
• Leader Election — coordination services are the most common implementation substrate for leader election in practice.
• Distributed Locking — coordination services provide the primitives (ephemeral nodes, leases, CAS) for correct distributed locks.

Further reading

• ZooKeeper Overview — Apache — official architecture documentation covering znodes, watches, and guarantees.
• etcd v3 API Documentation — comprehensive API reference for KV, lease, and watch operations.
• ZooKeeper: Wait-free coordination for Internet-scale systems — Hunt et al. — the original ZooKeeper paper.