Container Orchestration

What it is

Container orchestration automates the deployment, scaling, networking, and lifecycle management of containerised workloads across a cluster of machines. Kubernetes (K8s) is the dominant container orchestration platform. Its architecture consists of a control plane (the cluster brain) and a data plane (the nodes running workloads).

The control plane components: API Server — the central HTTPS API for all cluster operations; all tools (kubectl, GitOps agents, controllers) interact exclusively with the API server. etcd — distributed key-value store holding all cluster state; the single source of truth; critical to back up. Scheduler — assigns pods to nodes based on resource requests, affinity rules, taints/tolerations. Controller Manager — runs control loops ensuring actual state matches desired state (e.g., Deployment controller ensures the correct number of replicas are running). Cloud Controller Manager — integrates with cloud provider APIs for LoadBalancer Services, PersistentVolumes backed by EBS/GCE PD.

The data plane runs on each worker node: kubelet — agent that receives pod specs from API server and manages containers via the container runtime; kube-proxy — implements Kubernetes Service networking via iptables/IPVS rules; Container runtime — containerd or CRI-O runs containers from images. Key workload types: Deployment — stateless, rolling updates, horizontal scaling; StatefulSet — stateful apps needing stable network identities and ordered pod management (databases, Kafka); DaemonSet — runs one pod per node (log forwarders, monitoring agents, CNI plugins); Job/CronJob — batch and scheduled work.

Why it exists

Managing containers at scale without orchestration requires solving scheduling (which host runs which container?), service discovery (how do containers find each other?), scaling (how to add capacity under load?), health management (restart crashed containers, route around failed nodes), and rolling updates. Doing this with shell scripts and custom tooling at scale is the pre-Kubernetes reality many teams lived through. Kubernetes provides a declarative, API-driven model for all of this, backed by a large ecosystem and consistent behaviour across cloud providers.

When to use

• Microservices architectures running multiple services that benefit from consistent deployment, scaling, and networking abstractions.
• Workloads with variable load requiring autoscaling — HPA and KEDA provide mature event-driven and metric-driven scaling.
• Teams wanting portable, cloud-agnostic deployment that works identically on EKS, GKE, AKS, and on-premises clusters.

When not to use

• Simple single-service applications — Kubernetes adds substantial operational complexity; App Service, Cloud Run, or Fargate are better fits for single applications without microservices complexity.

Typical architecture

KUBERNETES DEPLOYMENT (stateless service):
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout, version: v1.3.0 }
    spec:
      # Spread across availability zones
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
      containers:
      - name: checkout
        image: ghcr.io/org/checkout:v1.3.0
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "1"
            memory: "512Mi"
        livenessProbe:
          httpGet: { path: /healthz, port: 8080 }
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet: { path: /ready, port: 8080 }
          initialDelaySeconds: 5
          periodSeconds: 10
      # Graceful shutdown
      terminationGracePeriodSeconds: 60

HORIZONTAL POD AUTOSCALER (HPA):
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 60 }
  - type: Pods
    pods:
      metric: { name: requests_per_second }
      target: { type: AverageValue, averageValue: "1000" }

KEDA (event-driven autoscaling):
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-processor
spec:
  scaleTargetRef:
    name: order-processor
  minReplicaCount: 0   # Scale to zero when queue empty
  maxReplicaCount: 100
  triggers:
  - type: aws-sqs-queue
    metadata:
      queueURL: https://sqs.eu-west-1.amazonaws.com/...
      targetQueueLength: "5"  # 1 pod per 5 queue messages

POD DISRUPTION BUDGET (availability guarantee):
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-pdb
spec:
  minAvailable: "60%"  # At least 60% of pods must be running
  selector:
    matchLabels: { app: checkout }
  # Prevents node drains from disrupting >40% of pods

Pros and cons

Implementation notes

Always set both requests and limits for CPU and memory. Requests determine pod scheduling (Kubernetes only schedules a pod to a node with sufficient available resources). Limits cap resource usage — memory limits cause OOMKilled, CPU limits cause throttling. Avoid setting CPU limits equal to CPU requests (this causes unnecessary throttling on bursty workloads); instead, set CPU limits 2-4x requests. Use the Vertical Pod Autoscaler (VPA) in recommendation mode to observe actual resource usage over time and right-size requests/limits based on real data rather than guessing.

Deploy PodDisruptionBudgets (PDBs) for all production workloads. Without PDBs, a node drain (for maintenance or node pool upgrade) can terminate all pods of a Deployment simultaneously if they happen to be on draining nodes. A PDB with minAvailable: 60% ensures that node drains proceed pod-by-pod, maintaining availability. For critical services, set PDB to minAvailable: N-1 where N is the minimum desired replica count, ensuring at least one pod is always terminating-not-yet-replaced during disruptions.

Common failure modes

• Missing resource requests: Pods without resource requests are scheduled on already-contended nodes; under load, they compete for CPU and are OOMKilled; cluster appears healthy by pod count but experiences widespread performance degradation.
• Readiness probe misconfiguration: Readiness probe succeeds too early (before app is truly ready) or fails too aggressively under load; traffic is sent to pods that are not ready, causing errors; or healthy pods are removed from service rotation during load spikes.
• Ignoring topology spread: All pods scheduled to the same availability zone; one AZ failure takes down the entire workload despite Kubernetes multi-AZ node groups; always configure topologySpreadConstraints for production workloads.

Decision checklist

• Are resource requests and limits set for all containers?
• Are liveness and readiness probes configured for all services?
• Are PodDisruptionBudgets deployed for production workloads?
• Are topologySpreadConstraints configured for multi-AZ distribution?
• Is HPA configured for services with variable load?
• Are StatefulSet-managed databases considered for migration to managed services?

Example use cases

• Event-driven batch processing: Order processing service uses KEDA with SQS trigger; scales from 0 pods at night to 80 pods during morning peak; scales back to 0 when queue empties; 90% cost reduction vs always-on fleet sized for peak.
• Zero-downtime cluster upgrade: EKS node pool upgrade drains nodes one at a time; PDBs ensure checkout service maintains >=3 running pods throughout; service health metrics show no error rate increase during 45-minute upgrade window.
• Multi-tenant platform: Platform team operates a single cluster with Namespace per team; ResourceQuotas limit each team's CPU/memory consumption; NetworkPolicies enforce namespace isolation; RBAC ensures teams can only access their own namespaces; teams deploy independently without platform team involvement.

Related patterns

• Service Mesh — mTLS, traffic management, and observability layer for inter-pod communication.
• GitOps — Declarative management of Kubernetes workloads from Git.
• Infrastructure as Code — Provisioning the Kubernetes cluster itself via Terraform.

Further reading

• Kubernetes Concepts Documentation
• KEDA Documentation
• Kubernetes Production Best Practices — learnk8s