Distributed Systems Reliability

Split Brain Problem

What happens when a network partition causes two nodes to each believe they are the sole primary — and how to prevent, detect, and recover from it.

⏱ 9 min read

What it is

Split brain is a condition in a distributed cluster where a network partition causes two or more nodes to each believe they are the authoritative primary for the same data, simultaneously accepting writes independently of each other. Both "brains" proceed as if the other does not exist, diverging in state. When the partition heals, the cluster must reconcile conflicting writes — a process that may be impossible without data loss or manual intervention for certain data types.

Why it exists

Leader election and primary selection algorithms rely on nodes being able to communicate. When a network partition splits a cluster into isolated sub-groups, each sub-group may independently detect the absence of a primary and elect one from its own members. The CAP theorem formalises this tension: during a partition, a system must choose between consistency (refusing writes until the partition is resolved) and availability (continuing to accept writes on both sides, risking divergence).

When to use

Understanding split brain is essential for designing any replicated stateful system — it is not an optional consideration.
High-consistency systems (financial data, inventory, user identity) must prevent split brain at all costs, even at the expense of availability.
Systems prioritising availability over consistency (social media feeds, analytics) may tolerate split brain with appropriate reconciliation.

When not to use

Not a pattern to avoid, but a failure mode to mitigate — every distributed system with replication must address this.

Typical architecture

Split Brain Scenario
═════════════════════

Before partition:
  [Primary: Node A] ←─── replication ───► [Replica: Node B]
                                           [Replica: Node C]

Network partition severs A from B and C:
  ┌─────────────────────┐   ║   ┌──────────────────────────┐
  │  Partition 1        │   ║   │  Partition 2             │
  │  Node A: "I am     │   ║   │  Node B + C detect A     │
  │  still primary"     │   ║   │  missing → elect B as    │
  │  accepts writes     │   ║   │  new primary → accepts   │
  └─────────────────────┘   ║   │  writes                  │
                            ║   └──────────────────────────┘

Result: conflicting writes on A and on B simultaneously.

Prevention: Quorum-based writes (W = majority)
  → Node A alone = 1 node  < majority of 3 → A cannot write
  → Node B + C   = 2 nodes = majority of 3 → B can write ✓

Pros and cons

Prevention strategies

Quorum writes: require W > N/2 acknowledgements; minority partition cannot form quorum.
Fencing tokens: storage rejects writes with stale epoch numbers even if old primary is still running.
External arbitrator (witness): third node or service acts as tiebreaker; minority without access to arbitrator cannot elect primary.
STONITH (Shoot The Other Node In The Head): uses out-of-band power management to forcibly terminate the suspected failed node before promoting a new primary.

Recovery challenges

Conflicting writes on non-commutative data (balance updates, inventory counts) cannot be auto-merged.
Last-write-wins resolution causes silent data loss.
Reconciliation may require manual intervention and data auditing.
Users may have acted on stale data visible during the split, creating external side effects that cannot be undone.

Implementation notes

Elasticsearch split brain was historically managed with the minimum_master_nodes setting (deprecated in 7.x, replaced by automatic quorum in 8.x). Setting this to ⌊N/2⌋+1 prevents a minority shard from electing a master. In Elasticsearch 8+, cluster bootstrapping and voting configurations handle this automatically.

MongoDB uses a replica set with a primary elected by majority vote. A partition that isolates the primary from the majority will step it down (it detects it can no longer reach a majority). The majority partition elects a new primary. The old primary becomes a secondary and rolls back any writes not replicated before the partition.

Redis Sentinel uses a quorum for the number of sentinels that must agree a master is down, and a separate "majority" requirement for the number of sentinels that must authorise a failover. Setting min-replicas-to-write on the master forces it to stop accepting writes if too few replicas are connected, preventing the split-brain window.

STONITH is used in traditional Linux HA clusters (Pacemaker/Corosync). When a node is suspected failed, the cluster initiates a STONITH (power-off via IPMI/DRAC/iLO) before promoting a standby, guaranteeing the old primary is dead before the new one starts writing. This prevents split brain at the cost of requiring physical/virtual power management infrastructure.

Common failure modes

Under-replication before partition: replica lag means the minority partition had a more up-to-date node, causing the promoted majority-side primary to start from an older state.
STONITH failure: STONITH command fails (network to IPMI controller is also down), leaving the cluster unable to safely promote a new primary.
Flapping partition: partition heals and re-occurs rapidly, causing repeated leadership changes and log corruption.
Application-level split brain: load balancer routes requests to both the old primary and the new primary during transition; application layer does not detect the ambiguity.

Decision checklist

Can the system tolerate temporary unavailability during a partition, or does it need to remain available on both sides?
Are write quorum requirements configured to prevent minority partitions from accepting writes?
Are fencing tokens used at the storage layer to reject stale-primary writes?
Is there a runbook for manual conflict resolution when automatic reconciliation is impossible?
Has the partition recovery procedure been tested in a chaos engineering exercise?
Is STONITH or equivalent forceful node termination available for correctness-critical systems?

Example use cases

MongoDB replica set during a data centre network outage — majority partition elects new primary, minority becomes read-only.
Elasticsearch cluster healing — detecting and rolling back divergent shards after a split.
PostgreSQL with Patroni — uses DCS (etcd/Consul) quorum; old primary demotes itself when it loses DCS connectivity.
Redis Sentinel failover — majority of sentinels required to authorise promotion of a replica.

Leader Election — split brain is the primary hazard that leader election algorithms must prevent.
Distributed Locking — fencing tokens (from the lock service) prevent old primaries from writing after re-election.
Quorum Reads & Writes — the primary technical mechanism for preventing split brain writes.