Reliability & Resilience Architecture

Failure Domains

Architectural boundaries that contain the blast radius of failures, preventing faults in one domain from propagating to others.

⏱ 10 min read

What it is

A failure domain is a group of components that share a common failure mode — when one component in the domain fails, all other components in the same domain are at risk of failing for the same underlying cause. Conversely, components in different failure domains are isolated from each other's failures by design: hardware boundaries (separate physical servers, racks, data centres), network boundaries (separate subnets, availability zones, regions), software boundaries (separate processes, pods, deployments), and dependency boundaries (separate databases, queues, external APIs). Failure domain analysis is the practice of identifying these groupings and deliberately architecting systems so that the maximum impact of any single failure is bounded to a single domain.

Failure domains exist at multiple layers of the infrastructure stack simultaneously: an AWS Availability Zone is a failure domain at the physical layer; a Kubernetes node is a failure domain at the compute layer; a database primary instance is a failure domain at the data layer; a microservice is a failure domain at the application layer; and a shared dependency (centralised auth service, shared Redis cluster) can inadvertently make multiple services part of the same failure domain even when they are deployed to separate infrastructure.

Why it exists

The goal of failure domain analysis is to make the maximum impact of any single failure commensurate with the size of the failure, not the size of the entire system. Without explicit failure domain design, a single hardware failure, configuration error, or bad deployment can cascade through the entire system via shared dependencies, shared infrastructure, or tightly coupled components. By designing explicit domain boundaries, each failure's blast radius is contained: a bad deployment to one canary cell affects 5% of traffic; an availability zone failure affects 33% of capacity; a region failure triggers regional failover with predetermined RTO/RPO targets. The system degrades gracefully in proportion to the scope of the failure rather than failing completely for any failure at any layer.

When to use

  • Every production system with a non-trivial availability requirement — failure domain analysis is a core architectural activity, not an advanced optimisation.
  • When designing multi-AZ or multi-region architectures — AZ/region boundaries must be actual failure domain boundaries, not just deployment topology.
  • Before deploying shared infrastructure (shared databases, shared caches, shared message brokers) — shared infrastructure creates shared failure domains.
  • When setting availability SLOs — the achievable SLO is bounded by the worst-case single failure domain's impact on end-to-end availability.
  • When performing blast radius analysis for chaos engineering experiments — experiments should respect failure domain boundaries.

When not to use

  • There is no scenario where failure domain analysis should be skipped — but the depth of analysis should be proportionate to the system's criticality and scale.
  • For low-criticality internal tools where any failure is acceptable, fine-grained domain isolation adds cost without meaningful benefit — keep the architecture simple.

Typical architecture

Failure Domain Hierarchy — AWS Multi-AZ Application
═══════════════════════════════════════════════════════════

REGION (us-east-1)  ←── region failure domain
│
├── AZ: us-east-1a  ←── AZ failure domain
│   ├── EC2 instances (ASG)
│   ├── RDS Primary
│   └── ElastiCache Primary
│
├── AZ: us-east-1b  ←── separate AZ failure domain
│   ├── EC2 instances (ASG)
│   ├── RDS Standby (Multi-AZ)
│   └── ElastiCache Replica
│
└── AZ: us-east-1c  ←── separate AZ failure domain
    └── EC2 instances (ASG)

Shared failure domain risks to identify and mitigate:
┌──────────────────────────────────────────────────────────────┐
│ Risk: All services share one database → DB is a failure       │
│       domain that spans all service boundaries.               │
│ Mitigation: Per-service databases, read replicas, circuit     │
│             breaker per service.                              │
│                                                               │
│ Risk: All AZs use same NAT Gateway → NAT is single point     │
│       of failure within the region.                           │
│ Mitigation: One NAT Gateway per AZ.                          │
│                                                               │
│ Risk: Config service is single AZ → config deployment is      │
│       a failure domain for all services that depend on it.    │
│ Mitigation: Multi-AZ config service with local caching.      │
└──────────────────────────────────────────────────────────────┘

Pros and cons

Pros

  • Bounds the maximum blast radius of any single failure — a correctly isolated failure domain cannot cascade to other domains.
  • Makes availability SLO calculations tractable — probability of domain failure × fraction of traffic in that domain = expected unavailability contribution.
  • Provides the architectural foundation for graceful degradation — each domain can be designed to degrade independently.
  • Enables independent deployment and upgrade within domains — a bad deployment to one cell affects only that cell's traffic fraction.
  • Aligns chaos engineering experiment design with real failure boundaries — experiments targeting failure domains produce the most realistic and useful results.

Cons

  • Strict domain isolation increases infrastructure cost — per-service databases, per-AZ NAT gateways, and per-domain shared resources are more expensive than shared infrastructure.
  • Cross-domain operations (distributed transactions, cross-AZ reads) add complexity and latency — domain isolation and data locality are in tension.
  • Dependency analysis is ongoing work — new dependencies and shared components continuously erode domain boundaries if not actively managed.
  • Over-isolation at fine-grained levels (per-pod failure domains) can make a system complex without proportionate reliability benefit.

Implementation notes

Draw failure domain boundaries as concentric zones in a dependency diagram and identify every cross-domain dependency. For each cross-domain dependency, ask: if the dependency fails completely, what is the impact on each dependent? The answer reveals hidden shared failure domains. Common cross-domain dependencies that collapse domain isolation: a shared database accessed by multiple microservices (fix: per-service databases or read replicas with circuit breakers); a centralised authentication service without caching (fix: local JWT validation with fallback to cached validation on auth service outage); a shared Kafka cluster (fix: circuit breakers, consumer-side backpressure, separate topic quotas per consumer group).

Design cell-based architectures for the highest criticality services: partition the user population into cells (by user ID range, geographic region, or tenant ID), deploy a full stack of services per cell, and ensure cells share no mutable state. A cell failure affects only the users assigned to that cell. This is the approach used by Amazon, Netflix, and Stripe for their most critical services — each cell is an independent failure domain.

Common failure modes

  • Shared dependency collapse: multiple microservices share a single dependency (database, cache, auth service) that becomes a single failure domain spanning all services — one dependency failure takes down all services simultaneously.
  • Cross-AZ single points: using a single NAT gateway, a single load balancer, or a single configuration service in one AZ while claiming multi-AZ resilience — the shared single-AZ component is a hidden failure domain.
  • Correlated deployments: deploying all cells or all AZs simultaneously in a rolling update — a bad deployment hits all cells before being detected, removing cell isolation as a protection against bad deployments.
  • Implicit shared state: microservices that appear independent but write to the same database tables or read from the same cache namespace — they are actually in the same failure domain.
  • Region-level shared components: a global configuration store, a global rate limiter, or a global authentication token cache that spans regions — a failure corrupts the global component and takes down all regions simultaneously.

Decision checklist

  • Have all failure domains been explicitly documented in an architecture diagram showing shared components?
  • Is the maximum impact of any single AZ failure bounded to <34% of capacity (assuming 3 AZs with even traffic distribution)?
  • Are all shared dependencies (databases, caches, auth services, config services) either multi-AZ or backed by a circuit breaker with local fallback?
  • Is there one NAT gateway per AZ (not one per region) to avoid cross-AZ NAT as a single failure domain?
  • Are new shared infrastructure components reviewed for failure domain impact before deployment?
  • Are deployments staggered across cells/canaries (not all-at-once) to preserve cell isolation as a protection against bad deployments?
  • Does the availability SLO calculation account for shared dependency failure modes, not just per-component availability?

Example use cases

  • AWS deploys each service across three AZs as independent failure domains with per-AZ NAT gateways, per-AZ load balancer nodes, and per-AZ database standbys — any single AZ failure causes at most 33% capacity loss, not a full outage.
  • Stripe's cell-based architecture partitions payment processing into independent cells by merchant ID range — a configuration error affecting one cell impacts a bounded fraction of merchants without affecting others.
  • A microservices architecture identifies that five independent services share a single Redis cluster for session caching — after failure domain analysis, each service is given a separate Redis namespace with its own circuit breaker, preventing a single Redis OOM from cascading across all five services.
  • Bulkhead Pattern — implements software-level failure domain isolation for concurrent workloads.
  • Circuit Breaker — provides the cross-domain protection needed when services in different failure domains must interact.
  • Chaos Engineering — experiments should be designed to respect failure domain boundaries as their blast radius.

Further reading