Horizontal Scaling

What it is

Horizontal scaling (also called "scaling out") is the practice of adding more machines or instances to a system to handle increased load, rather than upgrading the existing machine's capacity. Each new instance is a peer of the existing ones, collectively sharing the total workload. A load balancer sits in front of the fleet and distributes incoming requests across all available instances according to a chosen algorithm such as round-robin, least-connections, or IP hash.

Unlike vertical scaling, horizontal scaling is theoretically unbounded — you can keep adding instances as long as the workload can be parallelized. In practice, the shared state problem and coordination overhead impose a ceiling, but for stateless compute layers, true linear scalability is achievable. The pattern is foundational to every major cloud-native architecture and is the primary mechanism behind auto-scaling groups in AWS, GCP, and Azure.

Why it exists

Single-machine vertical limits were hit by web-scale workloads in the early 2000s. No matter how powerful a single server becomes, eventually the CPU or memory ceiling is reached, and a hardware upgrade requires downtime. Horizontal scaling sidesteps both problems: commodity hardware is cheap, instances can be added and removed without service interruption, and failures of individual nodes do not bring down the whole system. The economic argument is equally compelling — many small instances can deliver more aggregate capacity at lower cost than one massive server.

Auto-scaling takes the pattern further. Cloud platforms can monitor metrics such as CPU utilization, request queue depth, or custom business metrics and automatically launch or terminate instances to match demand precisely. This transforms capacity planning from a periodic manual exercise into a continuous automated feedback loop, enabling cost efficiency through right-sizing and providing elastic resilience against traffic spikes.

When to use

• Your application tier is stateless or can be made stateless by externalizing session and shared state.
• Traffic patterns are spiky or unpredictable and you need to scale up and down quickly without downtime.
• You require high availability and can tolerate the loss of individual instances without service disruption.
• Cost efficiency is important and you want to pay only for the compute you actually need at any given moment.
• Your workload is embarrassingly parallel — each request can be handled independently by any instance.
• You are running microservices where individual services have different resource profiles and need to scale independently.

When not to use

• The workload is inherently stateful and state cannot be externalized without unacceptable latency or complexity (e.g., long-running computational sessions with large in-memory state).
• The bottleneck is in a component that cannot be horizontally scaled — most commonly a single relational database write primary. Adding app-tier instances simply moves the bottleneck without solving it.
• Licensing costs scale per-instance, making a fleet of small instances more expensive than a few large ones.
• The orchestration and coordination overhead (service discovery, distributed tracing, consistent configuration) outweighs the scaling benefit for a small, stable workload.

Typical architecture

  Internet Traffic
        │
        ▼
  ┌─────────────────────┐
  │     Load Balancer    │  (ALB / NGINX / HAProxy)
  │  Round-robin / LC   │
  └──────┬──────┬───────┘
         │      │
   ┌─────┘      └─────┐
   ▼                  ▼
┌──────────┐     ┌──────────┐     ┌──────────┐
│  App #1  │     │  App #2  │ ... │  App #N  │
│ stateless│     │ stateless│     │ stateless│
└────┬─────┘     └────┬─────┘     └────┬─────┘
     │                │                │
     └────────┬────────────────────────┘
              ▼
   ┌─────────────────────┐
   │  Shared State Layer  │
   │  Redis / Memcached   │
   │  (sessions, cache)   │
   └─────────────────────┘
              │
              ▼
   ┌─────────────────────┐
   │   Persistent Store   │
   │  RDS / Aurora / PG   │
   └─────────────────────┘

Auto-Scaling Group policy:
  CPU > 70%  → +2 instances (scale-out)
  CPU < 30%  → -1 instance  (scale-in, cooldown 300s)
          

Pros and cons

Implementation notes

Statelessness is mandatory. Move HTTP session data to an external store (Redis with TTL-based expiry is the standard). Ensure no in-process state is mutated during request handling that must survive a process restart. Application configuration should be read at startup from environment variables or a configuration service, not embedded in instance-local files. Avoid any use of local disk for durable data; use object storage or a shared network mount instead.

Load balancer configuration: For most HTTP workloads, use an Application Load Balancer with least-connections or round-robin. Sticky sessions (session affinity by cookie) are a pragmatic fallback when externalized session storage is impractical, but they reduce load distribution quality and must be combined with graceful instance draining on scale-in. Health checks must be tuned carefully — too aggressive and healthy instances are cycled unnecessarily; too lenient and unhealthy instances continue receiving traffic. A typical configuration uses a 5-second interval, 2-second timeout, 2 consecutive failures to mark unhealthy, and 3 successes to return to rotation. Auto-scaling triggers: CPU is the bluntest metric. Prefer request-per-second or queue depth metrics for request-driven services, and custom business metrics for data processing pipelines. Add a warm-up period (typically 60–300 seconds) before a new instance starts receiving full traffic to avoid overloading the database connection pool during initialization.

Common failure modes

• Database connection storm: Auto-scaling adds 20 new instances simultaneously, each opening 50 connections, overwhelming the database max_connections. Mitigate with connection poolers (PgBouncer) and staggered scale-out.
• Sticky session imbalance: Session affinity pins too many users to a single instance, negating load distribution. Monitor per-instance request rates and set affinity timeout.
• Scale-in termination of in-flight requests: An instance is terminated before completing long-running requests. Implement graceful shutdown with a deregistration delay matching your P99 request duration.
• Split-brain configuration: Instances started at different times run different application versions if deployments are rolling and the rollback is incomplete. Use feature flags and backward-compatible schema changes.
• Shared cache stampede: After a cache flush, all instances simultaneously miss and hammer the database. Implement cache warming on startup and probabilistic early re-computation.

Decision checklist

• Application tier is stateless or a clear plan exists to externalize all session and in-memory state.
• A shared session store (Redis/Memcached) with HA configuration is provisioned or planned.
• Health check endpoints are implemented and return meaningful status including dependency checks.
• Graceful shutdown is implemented (SIGTERM handling, in-flight request drain) with appropriate deregistration delay.
• Auto-scaling policies are based on meaningful metrics (RPS, queue depth) not just CPU; scale-in cooldown periods are configured.
• Database connection pooling is in place to prevent connection exhaustion during rapid scale-out events.
• Deployment strategy (rolling, blue/green, canary) is chosen and the load balancer is configured to support it.

Example use cases

• E-commerce flash sales: An online retailer autoscales its product-listing service from 5 to 200 instances in minutes when a promotional event drives 40× normal traffic, then scales back down to save cost.
• SaaS API tier: A multi-tenant API platform runs a fleet of stateless API servers behind an ALB; each service independently autoscales based on request queue depth, allowing the user-service and the reporting-service to scale separately.
• Media transcoding workers: A video platform launches spot-instance worker pools to consume transcoding jobs from an SQS queue, scaling the fleet proportionally to queue depth and terminating idle workers automatically.

Related patterns

• Vertical Scaling — the alternative approach; useful for stateful workloads or as a complement when the instance size itself is the bottleneck.
• Caching Strategies — essential companion to reduce the shared database load that horizontal scaling exposes.
• Circuit Breaker — protects the shared state layer (Redis, DB) from being overwhelmed by a cascading failure across all instances.
• Service Discovery — needed when horizontally scaled services must locate each other dynamically rather than via static configuration.

Further reading

• AWS Auto Scaling for EC2 — Official Blog
• The Twelve-Factor App: Processes (Statelessness) — 12factor.net
• NGINX Load Balancing Overview