Cloud Architecture Scalability

Auto-Scaling Patterns

Mechanisms for automatically adjusting compute capacity — pods, nodes, or instances — in response to demand signals, enabling cost efficiency and resilience without manual intervention.

⏱ 11 min read

What it is

Auto-scaling is the capability of a system to increase or decrease its compute resources automatically in response to load signals. In cloud and Kubernetes environments, this operates across three dimensions: Horizontal Pod Autoscaler (HPA) scales the number of pod replicas within a Deployment or StatefulSet based on CPU utilisation, custom metrics, or external metrics; Vertical Pod Autoscaler (VPA) adjusts the CPU and memory requests/limits of individual pods based on historical usage patterns; and Cluster Autoscaler (CA) adds or removes EC2/GCE/Azure VM nodes from the cluster node group when pods cannot be scheduled due to resource constraints or when nodes are underutilised.

KEDA (Kubernetes Event-Driven Autoscaling) extends the HPA model to scale based on external event sources: SQS queue depth, Kafka consumer group lag, Azure Service Bus message count, Prometheus metrics, or any custom external scaler. KEDA can scale deployments to zero replicas when there are no events to process and from zero to N replicas when events arrive — enabling serverless-like cost efficiency for event-driven workloads running on Kubernetes. The combination of HPA, VPA (for recommendations), CA, and KEDA covers the full range of scaling patterns for container workloads.

Why it exists

Manual scaling — provisioning for peak load — wastes money 90% of the time when traffic is below peak. Provisioning for average load causes service degradation or outages during traffic spikes. Auto-scaling resolves this tension by making capacity a function of actual demand. For cloud workloads, the cost model is consumption-based, so idle compute directly increases bills. For microservices architectures with dozens of services, manually determining and adjusting replica counts would require constant engineering attention. Auto-scaling makes capacity management declarative: engineers specify the desired utilisation targets, and the scaling system continuously maintains them.

Predictive scaling extends reactive auto-scaling by using historical traffic patterns (time-of-day seasonality, day-of-week patterns) to pre-provision capacity before load arrives. This addresses the reactive lag problem: a traffic spike that takes 2 minutes to trigger a scale-out event and another 3 minutes for new instances to become healthy may still cause degraded service for the first 5 minutes. AWS Application Auto Scaling's predictive scaling uses ML to generate a scaling forecast and issues scale-out actions 5–10 minutes before predicted load increases.

When to use

  • Applications with variable, unpredictable traffic (consumer-facing APIs, e-commerce, media streaming) where over-provisioning for peak would waste significant resources.
  • Event-driven workloads (queue consumers, stream processors) where the workload volume is directly proportional to queue depth or topic lag — KEDA is purpose-built for this.
  • Kubernetes workloads where pod resource requests are miscalibrated — VPA can observe actual usage and recommend right-sized requests without manual tuning.
  • Batch workloads that burst periodically — autoscaling nodes in (Cluster Autoscaler) during job runs and scaling back to zero eliminates idle node costs between batch windows.
  • Multi-tenant SaaS platforms where each tenant workload may independently spike and needs isolated scaling without affecting other tenants.

When not to use

  • Stateful workloads with slow startup: Databases, Kafka brokers, and Elasticsearch nodes have multi-minute startup and data rebalancing processes that make reactive horizontal scaling impractical; scale these vertically or use pre-provisioned warm replicas.
  • In-place auto-scaling VPA with production pods: VPA in "Auto" mode evicts pods to apply new resource requests, causing brief disruptions; use VPA in "Recommend" mode for read-only right-sizing guidance without evictions.
  • Very short-duration spikes (sub-30-second): EC2 ASG or Kubernetes node provisioning takes 2–5 minutes; HPA pod scheduling takes 30–90 seconds; if your spike is shorter than your scale-out latency, pre-provisioned capacity or serverless is a better fit.
  • When startup initialisation is expensive: Services that require 5+ minutes of data loading or cache warming on startup may not benefit from reactive scaling; use warm pools or pre-scaled deployments during predictable events.

Typical architecture

Kubernetes Scaling Stack
────────────────────────────────────────────────────────
  External Metrics (Prometheus/Datadog/CloudWatch)
       │
       ▼
  KEDA ScaledObject                HPA
  (event-driven: SQS depth,        (CPU/memory/custom metrics)
   Kafka lag, etc.)                │
       │                           │
       └──────────┬────────────────┘
                  ▼
           Deployment / StatefulSet
              [pod] [pod] [pod] → scale replicas
                  │
                  ▼
           Cluster Autoscaler
           (pending pods → add node group instances)
           (underutilised nodes → cordon + drain + terminate)

EC2 Auto Scaling Group Patterns
────────────────────────────────────────────────────────
  Target Tracking Policy
    → maintain CPU at 60%
    → use ALB RequestCountPerTarget metric

  Warm Pool (for slow-starting instances)
    → pre-warmed instances in "Stopped" state
    → move to InService in seconds instead of minutes

  Mixed Instance Policy
    → 70% On-Demand (base) + 30% Spot (burst)
    → capacity-optimized Spot allocation

Pros and cons

Pros

  • Eliminates idle compute cost — resources scale down when demand falls, directly reducing cloud bills.
  • Provides resilience against traffic spikes without manual on-call intervention at 3 AM.
  • KEDA scale-to-zero eliminates idle cost entirely for event-driven workloads with infrequent bursts.
  • VPA recommendations expose over- and under-provisioned pods, improving resource efficiency cluster-wide.
  • Target tracking policies are self-correcting — no need to tune step scaling thresholds as workload character changes.

Cons

  • Scale-out latency (30 s to 5 min depending on layer) can cause degraded performance during sharp traffic spikes before new capacity becomes ready.
  • HPA and VPA cannot be used simultaneously on the same resource without conflict — HPA manages replicas, VPA manages requests; running both in Auto mode causes oscillation.
  • Aggressive scale-in policies can cause service disruption if pod drain periods are too short or PodDisruptionBudgets are misconfigured.
  • Cold start problem with KEDA scale-from-zero: the first events after a scale-to-zero period experience higher latency while pods initialise.
  • Cluster Autoscaler node removal decisions are complex with mixed workloads; misconfigured pod affinity can prevent scale-in even when nodes are underutilised.

Implementation notes

For HPA, start with CPU-based scaling using targetCPUUtilizationPercentage: 60 — 60% gives headroom for scale-out latency before saturation. For more responsive scaling, use custom metrics via Prometheus Adapter to expose application-level metrics (active connections, request queue depth) to the HPA. Configure behavior.scaleDown.stabilizationWindowSeconds: 300 to prevent flapping — this prevents immediate scale-in after a traffic burst subsides. Set minReplicas: 2 for any production workload to maintain redundancy; never scale to 1 replica for services that must survive a node failure.

For KEDA, install via Helm (helm install keda kedacore/keda) and define a ScaledObject pointing at your SQS queue with targetValue: 10 (messages per pod). Set minReplicaCount: 0 for true scale-to-zero and cooldownPeriod: 300 to avoid rapid scaling cycles. For the warm pool pattern on EC2 Auto Scaling: configure a Warm Pool with MinSize: 2 and state Stopped — instances are pre-initialised and can join the fleet in 30–60 seconds rather than 3–5 minutes. Combine with Lifecycle Hooks to run application warm-up scripts (cache loading, JVM JIT compilation) before the instance enters service.

Common failure modes

  • HPA thrashing: Scale-out triggers, new pods start (increasing available CPU), HPA scales in immediately — configure a stabilization window to prevent oscillation between scales.
  • Cluster Autoscaler unable to scale in: Pods without PodDisruptionBudgets, pods with local storage, or pods with strict node affinity prevent node draining; review CA logs for "node cannot be removed" reasons.
  • Missing resource requests: HPA uses CPU utilisation relative to requests — pods without CPU requests report 0% utilisation and HPA never scales; every pod must have resource requests defined.
  • Scale-out with downstream bottleneck: App tier scales out but database connection pool is exhausted — adding more pods amplifies downstream pressure; implement connection pooling (PgBouncer) before enabling aggressive HPA.
  • Cooldown period too short: EC2 ASG default 300 s cooldown; reducing it for faster scale-in can cause instances to terminate before in-flight requests complete — respect connection draining time.

Decision checklist

  • Do all pods have CPU and memory resource requests defined (required for HPA to function correctly)?
  • Is there a PodDisruptionBudget ensuring minimum available replicas during node draining?
  • Is your scale-out latency (pod start time) acceptable for your traffic spike profile?
  • Have you configured a scale-down stabilisation window to prevent flapping?
  • For KEDA scale-to-zero workloads, is the cold-start latency acceptable for the first message after an idle period?
  • Does your database or downstream dependency support the maximum concurrency that could result from a full scale-out event?

Example use cases

  • E-commerce Black Friday traffic: HPA scales the API tier from 10 to 80 pods in 4 minutes driven by ALB request count metric; predictive scaling pre-provisions 40 pods 15 minutes before midnight based on prior-year traffic pattern, eliminating the initial degradation window.
  • SQS-backed order processor: KEDA ScaledObject scales worker pods from 0 to 50 based on SQS queue depth; during quiet periods (midnight–6 AM) scale is zero, reducing cluster cost by 40% during those hours.
  • ML inference service: GPU node group uses Cluster Autoscaler; during model evaluation sprints, 20 GPU nodes spin up within 10 minutes driven by pending pod pressure, then scale back to 0 when jobs complete, costing $0 for compute between sprint cycles.

Further reading