Deployment Strategy Guide

The decision

How you deploy new versions of software determines the risk profile of every release, the speed of recovery from bad deployments, and the fidelity of your testing in production. The five primary strategies—blue-green, canary, rolling, recreate, and feature flags—represent very different trade-offs between deployment speed, infrastructure cost, rollback capability, and user impact. Combining strategies (feature flags inside a canary deployment, for example) allows fine-grained control that neither provides alone.

Deployment strategy is distinct from release strategy. A deployment physically replaces running software with a new version. A release makes functionality available to users. Feature flags decouple these two concerns, enabling you to deploy code that includes new features without releasing those features to users. This decoupling is the cornerstone of modern continuous deployment practices—code ships to production continuously, but features are released deliberately through flag toggles rather than deployment events.

The choice of deployment strategy depends on several factors: acceptable downtime window (zero vs. minutes vs. hours), infrastructure cost tolerance, application statefulness (stateless vs. session-affine), rollback time requirements (seconds vs. minutes), and traffic routing capabilities. Kubernetes provides first-class support for rolling and blue-green deployments; feature flag systems require separate infrastructure investment but provide the most flexibility for high-risk changes.

Why it matters

Deployment-related incidents are among the most common causes of production outages. A 2023 Puppet State of DevOps Report found that deployment failures account for roughly 20% of all production incidents. The deployment strategy directly controls the blast radius of a bad release: a recreate deployment (all instances replaced simultaneously) means a bad build immediately affects 100% of users, while a canary deployment can limit exposure to 1-5% of traffic, allowing detection and rollback before widespread impact. The difference between these outcomes can be the difference between a five-minute partial outage and a four-hour total outage.

Mean time to restore (MTTR) is strongly correlated with deployment strategy. Blue-green deployments can roll back in seconds by switching the load balancer back to the previous environment. Rolling deployments require reversing the rollout across all pods, which may take several minutes. Recreate deployments require a full redeployment of the previous version. For high-traffic applications where every minute of degradation costs revenue, the ability to execute an instant rollback is worth the infrastructure cost of maintaining a second environment.

The organizational dimension matters as much as the technical. Infrequent, high-risk "big bang" releases encourage risk aversion—teams hesitate to ship small improvements because the deployment overhead is the same regardless of change size. High-frequency, low-risk deployments via canary with feature flags encourage a culture of continuous improvement where small changes ship constantly and features toggle on when ready. Research consistently shows that high-deployment-frequency organizations have lower change failure rates, not higher, because small changes are easier to reason about and roll back.

Choose Blue-Green when

• Instant rollback (within seconds) is a hard requirement for the risk profile of the change
• The application has significant warm-up time (JVM class loading, cache warming) that makes gradual pod replacement impractical
• Database schema changes require both old and new application versions to run simultaneously during migration
• The infrastructure cost of a second identical environment is acceptable for the risk reduction provided
• Testing against full production traffic (by switching small traffic percentages to the new environment) is required before full cutover

Choose alternatives when

• Infrastructure cost of maintaining a duplicate environment is prohibitive—use rolling instead
• Gradual risk exposure with real user traffic monitoring is the priority—use canary
• Planned maintenance window allows brief downtime—use recreate for simplicity
• High-risk features need user-level control, independent of deployment—use feature flags
• The application is stateless and Kubernetes manages the fleet—use rolling for simplicity

Comparison

BLUE-GREEN DEPLOYMENT
═══════════════════════════════════════════════
Load Balancer ──┬──→ Blue  (v1, 100% traffic)
                └──→ Green (v2, idle)

Deploy v2 to Green → Test Green → Switch LB:
Load Balancer ──┬──→ Blue  (v1, idle, kept for rollback)
                └──→ Green (v2, 100% traffic)

Rollback: switch LB back to Blue (seconds)
Cost: 2x infrastructure while both environments live

CANARY DEPLOYMENT
═══════════════════════════════════════════════
Load Balancer ──┬──→ Stable (v1, 95% traffic)
                └──→ Canary (v2,  5% traffic) ← monitor

Watch metrics 15-60 min →
Load Balancer ──┬──→ Stable (v1, 50% traffic)
                └──→ Canary (v2, 50% traffic) ← monitor

Promote to 100% if healthy, or rollback to 0%

ROLLING DEPLOYMENT (Kubernetes default)
═══════════════════════════════════════════════
Pods: [v1][v1][v1][v1][v1][v1]
      ↓ replace one by one (maxUnavailable=1, maxSurge=1)
      [v2][v1][v1][v1][v1][v1]
      [v2][v2][v1][v1][v1][v1]
      ...
      [v2][v2][v2][v2][v2][v2]

Cost: minimal (1 extra pod during transition)
Rollback: set image back to v1 (takes minutes)

RECREATE DEPLOYMENT
═══════════════════════════════════════════════
Pods: [v1][v1][v1][v1][v1][v1]
      ↓ all terminated → downtime window ↓
Pods: [ ][ ][ ][ ][ ][ ]  ← DOWNTIME
      ↓ new pods start
Pods: [v2][v2][v2][v2][v2][v2]

Use when: state cannot run in mixed versions,
          downtime window is acceptable

FEATURE FLAGS (decouple deploy from release)
═══════════════════════════════════════════════
Deploy v2 (all code present, flag OFF)    → 0% users see new feature
Flag: new_checkout_flow = ON for 1% users → 1% users see new feature
Flag: new_checkout_flow = ON for 10%      → 10% users
Flag: new_checkout_flow = ON for 100%     → full release
Flag: new_checkout_flow = OFF             → instant rollback

Works with any deployment strategy above
Enables A/B testing, gradual rollout, kill switches
Requires: feature flag system (LaunchDarkly, Unleash, Flipt, AWS AppConfig)
          

Trade-offs

Implementation considerations

For Kubernetes rolling deployments, configure maxUnavailable and maxSurge to control the speed of rollout and resource overhead. Setting maxUnavailable: 0 and maxSurge: 1 ensures no capacity reduction during rollout—the new pod starts before the old one terminates—at the cost of one extra pod during the transition. Use minReadySeconds to ensure pods pass health checks before the rollout continues. Configure progressDeadlineSeconds to auto-fail a stalled rollout after a timeout, preventing a partial rollout from hanging indefinitely without alerting on-call.

For canary deployments, automated promotion based on SLOs is more reliable than manual observation. Tools like Argo Rollouts and Flagger integrate with Prometheus metrics to automatically advance canary traffic percentage when error rate and latency stay within bounds, or roll back when they exceed thresholds. A typical configuration: promote from 5% to 20% to 50% to 100% at 15-minute intervals, rolling back automatically if the error rate exceeds 0.1% or p99 latency increases by more than 50%. This removes human judgment from the hot path of a rollout and ensures consistent, data-driven decisions.

For feature flags, invest in a system that provides: real-time flag evaluation without application restart, targeting rules by user segment (beta users, internal employees, geographic region), percentage rollout with consistent user bucketing (the same user always sees the same variant), and an audit log of flag changes. Open-source options include Unleash (self-hosted), Flipt, and GrowthBook. Commercial options include LaunchDarkly and Split.io. The critical operational practice is flag hygiene: every feature flag must have a defined lifecycle and removal date. Flags that live indefinitely become technical debt—accumulations of conditional code paths that make the system harder to reason about.

Common mistakes

• Blue-green with stateful sessions: If the old (blue) environment holds active user sessions and traffic switches to green, users lose session state unless sessions are stored externally (Redis, database); always externalize session state before adopting blue-green.
• Canary percentage too small: A 1% canary on an endpoint with 10 requests per minute means 0.1 requests per minute to the canary—insufficient to detect low-frequency errors; size canary percentages to get statistically meaningful signal within your monitoring window.
• Rolling back without reversing DB migrations: If a deployment includes a database schema migration and the rollout is rolled back, the database schema may be incompatible with the old application version; always design migrations to be backward compatible (additive changes, not destructive ones).
• Feature flags without cleanup: Flags that are never removed accumulate as conditional branches throughout the codebase, increasing cognitive overhead and testing complexity; define a maximum flag lifetime policy and include flag removal in the release checklist.
• No health check configuration: Rolling deployments without readinessProbe and livenessProbe configured will route traffic to pods that are not ready, causing request failures during the rollout; always configure Kubernetes health probes.

Decision checklist

• What is the acceptable downtime window? Zero downtime rules out recreate; near-zero favors blue-green or canary.
• What is the rollback time requirement? Seconds favor blue-green; minutes are acceptable for rolling.
• Is the application stateless, or does it have warm-up requirements that make pod-by-pod replacement impractical?
• Does the deployment include database schema changes? If yes, ensure the schema change is backward compatible with both old and new application versions during the transition.
• Can the infrastructure cost of a second blue-green environment be justified by the risk profile?
• Do you have traffic routing infrastructure (Istio, ALB weighted routing) to support canary percentage control?
• Are high-risk features better decoupled from the deployment using feature flags?

Real-world examples

• Amazon (canary + automated rollback): Amazon's deployment infrastructure automatically canaries every deployment to a small fraction of production traffic and rolls back automatically when error rates exceed a threshold. This system—described in their well-architected framework—enables thousands of deployments per day across Amazon's services with automated safety guarantees rather than human observation windows.
• Facebook (feature flags): Facebook's Gatekeeper system allows engineers to deploy code with new features present but flagged off, then gradually roll features to percentages of users via the flag system rather than via deployment. This enables rapid deployment frequency while maintaining control over which users experience new features—decoupling the organizational risk of feature releases from the technical risk of deployments.
• Netflix (Spinnaker + canary analysis): Netflix open-sourced Spinnaker, their continuous delivery platform, which includes automated canary analysis (ACA) using statistical comparison of canary and baseline metrics. Netflix's deployment philosophy treats every deployment as a hypothesis about production behavior that should be validated with real traffic before full rollout.

Related decisions

• Cloud Provider Selection — cloud provider deployment mechanisms differ significantly (ECS, EKS, App Service)
• Deployment — CI/CD pipelines, GitOps, and infrastructure automation in depth
• Reliability — how deployment strategy interacts with availability and incident response
• Observability — monitoring and alerting required to safely drive canary deployments

Further reading

• Kubernetes Deployments — kubernetes.io Documentation
• Argo Rollouts: Kubernetes Progressive Delivery — argoproj.github.io
• Feature Toggles (aka Feature Flags) — Pete Hodgson, MartinFowler.com