Distributed Systems Consistency

Distributed Transactions

Coordinating atomic operations across multiple independent data stores — the mechanics, the costs, and the modern alternatives.

⏱ 10 min read

What it is

A distributed transaction is an operation that spans multiple independent resource managers (databases, message brokers, file systems) and must satisfy ACID properties — atomicity, consistency, isolation, and durability — across all of them. Either all participants commit their changes, or all roll back, with no partial outcomes visible to observers. The canonical protocol for achieving this is Two-Phase Commit (2PC), standardized as the X/Open XA specification.

Why it exists

Single-node databases achieve atomicity cheaply using write-ahead logs and local locks. Once data lives in separate systems — two relational databases, a database and a message queue, or databases in separate microservices — there is no single transaction log. Without coordination, a crash mid-operation leaves some participants committed and others not, creating permanent data inconsistency. Distributed transactions were introduced to provide the same atomicity guarantees engineers relied on in monolithic systems, even after distributing state across multiple nodes or services.

When to use

  • Resources are under the same vendor or infrastructure (e.g., two Oracle databases accessed via XA through the same transaction manager).
  • Correctness requirements are strict and financial/legal consequences of partial failures are severe (banking ledgers, payment settlement).
  • Participants already support XA (Java EE application servers, JMS brokers with XA support).
  • The transaction scope is small (two to three participants) and latency is acceptable.
  • You operate a tightly controlled private network where coordinator failure is manageable.

When not to use

  • Microservices with separate databases owned by separate teams — XA couples services at the infrastructure level.
  • High-throughput systems where 2PC lock hold times create throughput bottlenecks.
  • Cloud-native architectures spanning services from multiple vendors or cloud regions.
  • Services using NoSQL stores that do not implement XA protocols.
  • Systems that must tolerate coordinator node failure without extended blocking.

Typical architecture

Two-Phase Commit operates in two rounds of communication between a coordinator and N participants:

Phase 1 — Prepare
─────────────────
Coordinator                Participant A      Participant B
    │──── PREPARE ──────────────►│                │
    │──── PREPARE ───────────────────────────────►│
    │◄─── VOTE YES ─────────────│                │
    │◄─── VOTE YES ──────────────────────────────│

Phase 2 — Commit (all voted YES)
──────────────────────────────────
    │──── COMMIT ───────────────►│                │
    │──── COMMIT ────────────────────────────────►│
    │◄─── ACK ──────────────────│                │
    │◄─── ACK ────────────────────────────────────│

Phase 2 — Abort (any voted NO / coordinator timeout)
──────────────────────────────────────────────────────
    │──── ROLLBACK ─────────────►│                │
    │──── ROLLBACK ──────────────────────────────►│

XA is the standard C API / Java interface (JTA) that maps this protocol to specific resource managers. A transaction manager (e.g., Atomikos, Narayana, IBM TXSeries) drives the protocol and maintains a durable transaction log to recover after crashes.

Pros and cons

Pros

  • True atomicity across heterogeneous data stores without application-level compensation logic.
  • Well-understood protocol with decades of production usage in enterprise systems.
  • Supported by most relational databases and enterprise messaging systems via XA.
  • Integrates naturally with existing ACID semantics developers already understand.

Cons

  • Coordinator is a single point of failure — crash during Phase 2 leaves participants blocked holding locks indefinitely (the "blocking" problem).
  • Locks held across network round trips reduce throughput significantly at scale.
  • 3PC avoids blocking but is rarely implemented due to increased complexity and still not partition-tolerant.
  • Tight coupling between services — schema and infrastructure changes in one participant affect all.
  • Poor fit for polyglot or cloud-native environments where XA support is absent or unreliable.

Implementation notes

The coordinator failure problem is the central challenge: if the coordinator crashes after collecting all YES votes but before sending COMMIT, participants are stuck in the prepared state holding locks and cannot determine the outcome without the coordinator recovering. Three-Phase Commit (3PC) adds a pre-commit phase to reduce blocking but still cannot handle network partitions (by the FLP impossibility result, no deterministic algorithm can). In practice, durable coordinator logs and automated recovery are the standard mitigation. Most modern Java EE / Jakarta EE containers use Narayana or Atomikos as JTA transaction managers; Spring's JtaTransactionManager delegates to them. Ensure XA timeouts are tuned — default values are often far too long for interactive workloads.

Common failure modes

  • Coordinator crash in Phase 2: participants hold prepared-state locks indefinitely until coordinator recovers.
  • Participant timeout: participant unilaterally aborts after waiting too long for Phase 2 decision, causing inconsistency if others already committed.
  • Heuristic decisions: after timeout, participants may make unilateral commit/abort decisions (heuristic outcomes), which may diverge from the coordinator's intent.
  • Network partition isolating coordinator: partition prevents coordinator from delivering COMMIT to a subset of participants.
  • Transaction log loss: if coordinator log is lost before recovery, in-doubt transactions remain unresolved permanently.

Decision checklist

  • Do all participating stores support XA / two-phase commit protocols?
  • Is coordinator failure recovery acceptable (restart time, lock hold duration)?
  • Are the performance implications of distributed locks acceptable at your transaction rate?
  • Have you considered the Saga pattern as a lower-coupling alternative?
  • Is a transaction manager (Narayana, Atomikos) already in your stack, or does adding one increase operational burden?
  • Are network latencies between participants low and predictable?

Example use cases

  • Bank-to-bank transfer updating two Oracle databases in the same data centre via XA.
  • J2EE order processing system coordinating a relational database and a JMS message queue.
  • Mainframe financial batch jobs using CICS or IMS distributed transaction support.
  • SAP systems using XA to coordinate ERP database writes with connected systems.
  • Saga Pattern — the primary microservices alternative to distributed transactions, using compensating transactions instead of 2PC.
  • Compensating Transactions — semantic undo operations that replace atomic rollback in eventually-consistent workflows.
  • Distributed Locking — related coordination primitive; 2PC participants hold locks during prepared phase.

Further reading