Observability in Chaos Engineering

What it is

Chaos engineering is the practice of deliberately injecting failures into production or production-like systems to verify that they behave as expected under adverse conditions. Netflix coined the term and pioneered the discipline with Chaos Monkey. The critical dependency between chaos engineering and observability is bidirectional: observability is the prerequisite for safe chaos experiments (without rich telemetry, you cannot tell if an experiment is having unintended effects), and chaos engineering validates the adequacy of your observability (experiments reveal blind spots — failure modes that occur without generating useful signals in your dashboards or traces).

Every chaos experiment follows a structured process: (1) Hypothesis definition: "When one availability zone becomes unavailable, checkout success rate will remain above 99%" — a specific, measurable statement about expected system behaviour. (2) Steady-state baseline: Measure SLIs (error rate, latency, throughput) before the experiment to establish the baseline for comparison. Observability tooling must be in place before any experiment begins. (3) Blast radius control: Start small — inject failures affecting 1% of traffic or 1 instance before scaling. Define stop conditions (if error rate exceeds X, abort automatically). (4) Inject failure: Kill an instance, introduce network latency, corrupt a dependency response, exhaust resources. (5) Observe and measure: Compare current SLIs against baseline using your distributed tracing, metrics, and logs. (6) Restore and analyse: Stop the experiment, verify recovery, document findings and any gaps in observability discovered.

The tools used for chaos experiments vary by scope: Chaos Monkey (Netflix) randomly terminates EC2 instances in the Auto Scaling Group — simple, production-proven. Gremlin is a commercial platform with a rich catalogue of failure types (CPU, memory, network, disk, process, time, DNS), granular blast radius controls, and GameDay scheduling. LitmusChaos is a CNCF project that runs chaos experiments as Kubernetes ChaosEngine resources, integrates with Argo Workflows, and has hundreds of pre-built experiment templates for Kubernetes environments.

Why it exists

Traditional reliability testing — unit tests, integration tests, load tests — validates that systems work correctly under expected conditions. Chaos engineering validates that systems degrade gracefully under unexpected but inevitable conditions. Production systems experience hardware failures, network partitions, dependency degradation, and resource exhaustion that no test environment perfectly replicates. Netflix found that teams who ran regular chaos experiments had lower mean time to recovery (MTTR) in real incidents because they had practised the failure scenarios and had confidence in their observability. Without chaos engineering, teams often discover their monitoring blind spots and missing circuit breakers during real customer-impacting incidents rather than controlled experiments.

When to use

• Systems with mature observability — chaos experiments without telemetry are dangerous and unproductive; you cannot observe failure effects you cannot measure.
• Services with explicit SLOs — experiments validate that SLOs hold under failure conditions.
• Before major events (Black Friday, product launches) — GameDays verify system readiness.

When not to use

• Systems without sufficient observability — establish metrics, tracing, and alerting first; running chaos experiments without telemetry means you cannot tell when an experiment has unexpected effects, making it unsafe.
• Teams in early stages — chaos engineering is a reliability practice for mature systems; premature adoption diverts effort from more impactful reliability investments.

Typical architecture

CHAOS EXPERIMENT WORKFLOW:

1. BASELINE MEASUREMENT
   Before experiment, record:
   - Checkout success rate: 99.95%
   - p99 latency: 180ms
   - Error budget remaining: 95%

2. LITMUSCHAKOS EXPERIMENT DEFINITION
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-kill-experiment
spec:
  appinfo:
    appns: production
    applabel: app=checkout
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      components:
        env:
        - name: TOTAL_CHAOS_DURATION
          value: "60"        # 60 second experiment
        - name: CHAOS_INTERVAL
          value: "10"
        - name: FORCE
          value: "false"
        - name: PODS_AFFECTED_PERC
          value: "20"        # Kill 20% of pods

3. HYPOTHESIS VALIDATION (automated)
   After experiment:
   success_rate = query("checkout_success_rate[5m]")
   if success_rate < 0.99:
     result = "HYPOTHESIS FAILED"
     file_issue(severity="high")
   else:
     result = "HYPOTHESIS CONFIRMED"

4. OBSERVABILITY DURING EXPERIMENT
   Monitor in real-time:
   - Error rate vs baseline
   - Pod restart count
   - Request routing (was traffic redistributed?)
   - Circuit breaker state
   - Database connection pool pressure

BLAST RADIUS ESCALATION:
  Week 1: Kill 1 pod (5% of capacity)
  Week 2: Kill 20% of pods
  Week 3: Simulate AZ failure (all pods in 1 AZ)
  Quarter: Full GameDay (multi-failure scenario)

AUTOMATED STOP CONDITIONS (Gremlin):
  abort if error_rate > 2%
  abort if p99_latency > 1000ms
  abort if error_budget_remaining < 10%

Pros and cons

Implementation notes

Start with a Game Day in staging or pre-production rather than production. This allows the team to practice the experiment workflow, validate observability coverage, and build confidence before experimenting in production. In staging, deliberately misconfigure a circuit breaker or remove a retry policy before the experiment to verify that the experiment correctly surfaces these gaps. If staging experiments pass but production behaves differently, the staging environment fidelity is insufficient for the experiment to be meaningful.

Automate abort conditions in your chaos tooling. Gremlin and LitmusChaos both support automated halt criteria. Set these to trigger when SLIs exceed pre-agreed thresholds (typically 2× the normal error rate or 2× the normal p99 latency). Automated abort ensures that an experiment that goes wrong does not cause extended customer impact while the team notices and manually intervenes. Integrate chaos experiment results into a central "reliability scorecard" — which hypotheses passed, which failed, what gaps were found, what was remediated. This creates organisational visibility into reliability investment and progress.

Common failure modes

• Experiments without baselines: Running an experiment without first measuring steady-state SLIs means there is no comparison point — you cannot determine whether the experiment affected the system or not.
• Insufficient blast radius control: Starting with an experiment that affects 100% of capacity rather than gradually escalating; a well-designed experiment should affect the minimum blast radius needed to test the hypothesis.
• No observability before chaos: Running chaos experiments without distributed tracing and metrics means the experiment generates effects that cannot be observed — the experiment produces no actionable information.
• One-time GameDays: A single annual GameDay is insufficient; the most value comes from regular, automated, smaller experiments that continuously validate resilience as the system changes.

Decision checklist

• Is observability mature enough to safely run experiments (metrics, traces, alerts in place)?
• Is each experiment preceded by steady-state baseline measurement?
• Are automated abort conditions configured for all production experiments?
• Are experiments started at minimum blast radius and escalated gradually?
• Are hypothesis results tracked and failures used to drive reliability improvements?
• Is experiment scheduling coordinated with on-call rotation to ensure appropriate coverage?

Example use cases

• Circuit breaker validation: Hypothesis: "When the recommendation service returns 100% errors, the homepage continues to load without it." Experiment: inject 100% error responses from recommendation service using Gremlin. Outcome: homepage error rate spikes to 60% — circuit breaker was not configured for this dependency. Fix: add circuit breaker; re-run experiment: homepage error rate 0%, recommendations show fallback content.
• AZ failure GameDay: Quarterly exercise simulates complete loss of one AWS availability zone; teams across engineering validate that services with multi-AZ deployment correctly redistribute traffic; discovers that one service's connection pool does not drain failed connections when an AZ goes down, causing 5% error rate for 3 minutes; connection pool configuration fixed before the next real AZ event.
• Observability gap discovery: LitmusChaos kills 30% of payment service pods; expected alert (pod count below minimum) does not fire; investigation reveals the alert was misconfigured with wrong label selector; gap found and fixed in a controlled context rather than during a real incident.

Related patterns

• Three Pillars of Observability — The telemetry prerequisite for safe chaos experiments.
• Alerting Strategies — Experiments validate alerting coverage and correctness.
• Reliability & Resilience — Circuit breakers, bulkheads, and the patterns chaos experiments test.

Further reading

• Principles of Chaos Engineering
• LitmusChaos Documentation
• Gremlin: Chaos Engineering Platform