Cloud Architecture Reliability

Disaster Recovery Patterns

The four cloud DR strategies — from cheap-but-slow backup-restore to expensive-but-instant multi-site active-active — and how to choose between them based on your RTO and RPO requirements.

⏱ 11 min read

What it is

Recovery Time Objective (RTO) is the maximum acceptable duration of downtime following a disaster — the time from incident declaration to service restoration. Recovery Point Objective (RPO) is the maximum acceptable amount of data loss measured in time — if RPO is 1 hour and a disaster strikes at 14:00, the system must be restorable to its 13:00 state with no more than 1 hour of transaction loss. These two metrics are the primary inputs to DR strategy selection: lower RTO and RPO require more expensive, always-hot standby infrastructure; higher tolerance enables simpler, cheaper backup-based approaches.

AWS defines four DR strategies along a cost-vs-recovery-speed spectrum. Backup and Restore (RTO: hours, RPO: hours) stores backups in S3 and restores from scratch; cheapest option (~$50–200/month for backup storage). Pilot Light (RTO: 10–60 min, RPO: minutes) keeps a minimal version of the environment running in the DR region — just the database replication and core networking, with compute scaled to zero; cost ~$200–500/month. Warm Standby (RTO: minutes, RPO: seconds) runs a scaled-down but fully functional version of the system in the DR region; cost ~$500–3,000/month. Multi-Site Active-Active (RTO: near-zero, RPO: near-zero) runs full capacity in both regions simultaneously with traffic split between them; cost: 2× normal infrastructure.

Why it exists

Regional AWS failures occur — us-east-1 had a significant disruption in December 2021, and smaller AZ-level failures occur several times per year. Without a DR strategy, a regional failure means total service outage until AWS restores the region. For organisations with revenue-generating applications, the financial impact of downtime can be enormous: Amazon's own estimate of $220,000/minute for their systems implies that a 2-hour outage costs $26M. More relevant: a 4-hour outage for an e-commerce platform with $1M/day GMV costs $165K in direct revenue loss plus customer trust damage. The cost of warm standby DR ($1,000/month) is negligible compared to a single outage event.

DR strategy selection requires honesty about the business impact of downtime. Many organisations claim an RTO of "15 minutes" without building or testing the infrastructure required to achieve it. The technical requirement for 15-minute RTO is a Warm Standby with pre-provisioned compute in the DR region, automated health checks triggering DNS failover, and a runbook that has been tested in a GameDay exercise within the last 90 days. "We have backups in S3" does not deliver 15-minute RTO — it delivers 4–12 hour RTO depending on database size and restore automation maturity.

When to use

  • Backup and Restore: Internal tools, development environments, non-revenue-critical applications, or systems where hours of downtime and data loss are acceptable; budget-constrained projects with regulated data retention requirements.
  • Pilot Light: Small to mid-size applications where RTO of 30–60 minutes is acceptable and the cost of full standby is unjustified; database-heavy applications where data replication (Aurora Global Database, DynamoDB Global Tables) is the primary DR investment.
  • Warm Standby: Production applications with business-critical data where 15-minute RTO and near-zero RPO are required; regulated industries (financial services, healthcare) with formal DR requirements.
  • Multi-Site Active-Active: Global applications where any regional latency is unacceptable, or where business continuity requirements mandate zero downtime (critical payments infrastructure, real-time systems); organisations already operating in multiple regions for latency reasons.

When not to use

  • Active-Active for stateful monoliths: Active-active requires distributed, conflict-resolution-aware data management — forcing a monolithic application with a single-primary database into active-active creates data conflicts and split-brain risks that are worse than a clean failover.
  • Pilot Light without automated failover: A Pilot Light strategy where failover requires manual DNS changes and scaling operations does not deliver its advertised RTO in a real disaster where the on-call engineer is under stress at 3 AM.
  • Multi-site active-active for low-traffic applications: Paying 2× infrastructure costs for an internal application with 50 users and $50K annual revenue is rarely justifiable; match DR investment to business impact.

Typical architecture

DR Strategy Comparison
─────────────────────────────────────────────────────────────
Strategy         RTO        RPO        Cost/month   Always On
─────────────────────────────────────────────────────────────
Backup/Restore   Hours      Hours      $50–200      Backups only
Pilot Light      10–60 min  Minutes    $200–500     DB replication
Warm Standby     Minutes    Seconds    $500–3K      Scaled-down
Multi-Site AA    Near zero  Near zero  2× prod      Full capacity

Warm Standby Architecture (AWS)
─────────────────────────────────────────────────────────────
  Primary (us-east-1)           DR (eu-west-1)
  ─────────────────             ────────────────
  ALB  (active)                 ALB  (standby)
  ECS 10 tasks  ──┐             ECS  2 tasks (scale to 10 on failover)
  Aurora Primary  │  replication Aurora Read Replica
  ElastiCache     │  (async)    ElastiCache (pre-warmed)
                  └─────────────►
  Route 53 Failover Routing:
    Primary record → us-east-1 ALB (health check every 30s)
    Secondary record → eu-west-1 ALB (failover)

  On failover:
  1. Route 53 health check fails primary
  2. DNS flips to eu-west-1 (TTL: 30s)
  3. Aurora Global DB promotes eu-west-1 replica (~1 min)
  4. ECS scales from 2 → 10 tasks (~3 min)
  Total RTO: ~5 minutes

Pros and cons

Pros

  • Cloud DR is dramatically cheaper than traditional data centre DR — warm standby at $1,000/month vs $100K+ for a co-located secondary data centre.
  • Aurora Global Database replication achieves typical RPO of under 1 second across regions with no application changes.
  • Infrastructure-as-Code (CDK/Terraform) enables identical environment provisioning in DR region with version-controlled runbooks.
  • AWS Elastic Disaster Recovery (DRS) automates block-level replication and failover orchestration for lift-and-shift DR scenarios.
  • DR testing can be automated as part of quarterly GameDay exercises without customer impact using read-only chaos simulations.

Cons

  • Untested DR plans fail at the worst possible time; organisations that document RTO/RPO without running DR tests create false confidence.
  • Active-active requires conflict resolution strategy for writes in both regions — eventually consistent or last-write-wins semantics are application concerns the platform cannot solve automatically.
  • Cross-region data transfer costs for database replication are ongoing ($0.02/GB) and can be significant for high-write workloads.
  • Warm standby database promotion time for Aurora Global Database (1–2 minutes) and DNS propagation (30–60 seconds) are lower bounds on achievable RTO that are often underestimated.
  • DR environments drift from production over time without automated drift detection (Config Rules, Terraform plan CI).

Implementation notes

DR runbooks must be executable by any engineer on the on-call rotation, not just the architect who designed the system. Document every manual step with exact CLI commands, console URLs, and expected output. Automate what can be automated: Aurora Global Database promotion is a single API call (PromoteReadReplicaDBCluster); ECS service scaling is a single API call; Route 53 record updates are a single API call. Build a "break glass" Lambda or Step Functions state machine that executes all failover steps in sequence and sends status updates to an incident Slack channel. The ideal failover is triggered by a single command and completes in under 5 minutes without manual intervention.

DR testing cadence should be quarterly at a minimum for Tier 1 applications. A DR test should simulate the actual failure: terminate all instances in the primary region (or suspend Route 53 health checks), verify the DR environment serves traffic correctly, validate data integrity, and measure the actual RTO achieved. Document the measured RTO vs the target RTO, and update the runbook for any gaps discovered. For GameDay testing: use AWS FIS (Fault Injection Simulator) to inject region-level failures in a controlled environment without customer impact. Require each application team to pass a DR GameDay before receiving a production-readiness certification.

Common failure modes

  • Undocumented DR strategy: Leadership believes RTO is "1 hour" based on a conversation 2 years ago; no runbook exists; no test has been run; the actual RTO when needed is 8+ hours; run a DR test immediately to establish a baseline.
  • Database not replicated to DR region: Backups are in S3 Standard in us-east-1 (the same region that failed); restoring from a backup in the failed region requires the region to be functional; always replicate backups and snapshots cross-region.
  • Secrets not available in DR region: Application fails to start in DR region because Secrets Manager secrets are not replicated; use Secrets Manager cross-region replication or store secrets in both regions; test the DR environment boots successfully quarterly.
  • DNS TTL too high: Route 53 failover occurs but DNS record TTL is 300 seconds (5 minutes); downstream clients cache the old IP for up to 5 minutes after failover; set TTL to 30–60 seconds for failover records at all times, not just during incidents.
  • Application state not DR-aware: In-memory session state stored in a single-region ElastiCache cluster is lost on regional failover; use DynamoDB or Elasticache Global Datastore for session state that must survive regional failures.

Decision checklist

  • Have business stakeholders formally agreed on RTO and RPO targets, documented in the system's architecture decision record?
  • Are database backups and snapshots replicated cross-region to the DR target region?
  • Has the DR runbook been tested end-to-end in the last 90 days, with actual RTO and RPO measured and documented?
  • Is the DR environment provisioned from the same IaC code as production, with automated drift detection?
  • Are Secrets Manager secrets and Parameter Store parameters replicated to the DR region?
  • Is the failover process automated to the point where any on-call engineer can execute it in under 15 minutes, including DNS cutover and database promotion?

Example use cases

  • Financial services platform (Warm Standby): Regulatory requirement of RTO ≤ 15 minutes; Aurora Global Database replicates with 1-second lag; ECS warm standby with 2 tasks; quarterly GameDay confirms 6-minute measured RTO; $2,400/month DR cost vs $500K fine for regulatory non-compliance.
  • E-commerce checkout (Active-Active): Revenue impact of any downtime is $5K/minute; checkout service deployed in us-east-1 and eu-west-1 with DynamoDB Global Tables for cart data; Route 53 latency routing splits traffic between regions; any region failure results in 100% traffic to the healthy region within 30 seconds; zero-downtime failover validated monthly.
  • Internal analytics platform (Backup/Restore): 4-hour RTO and 24-hour RPO accepted by business; daily Aurora snapshots cross-replicated to us-west-2; restore runbook documents 3-hour restoration process; $80/month DR cost for a non-revenue-critical system.
  • Cloud Networking — Route 53 failover routing and multi-region networking underpin all cloud DR strategies.
  • Reliability & Resilience — DR is one pillar of broader reliability engineering; circuit breakers and bulkheads address intra-region failures.
  • Landing Zones — DR accounts and VPCs are part of the multi-account landing zone structure.

Further reading