Chaos engineering is the discipline of experimenting on a production system to build confidence in its ability to withstand turbulent conditions. Netflix pioneered this approach with Chaos Monkey, and it has become a standard practice at companies running distributed systems at scale. This guide covers chaos engineering principles, tools, and practical implementation — essential knowledge for SRE and system design interviews.
Why Chaos Engineering Exists
Distributed systems fail in complex, unpredictable ways. Traditional testing (unit tests, integration tests, load tests) cannot reproduce the emergent failures that occur when multiple components interact under real-world conditions. Examples: a network partition between service A and service B causes a cascade that takes down service C (which depends on neither A nor B) because of a shared connection pool. A DNS TTL expiry combined with a load balancer health check race condition causes 30 seconds of dropped traffic every 48 hours. A garbage collection pause on one node causes a Raft leader election timeout, which triggers a cluster rebalance, which overloads the remaining nodes. These failures are discovered in production during incidents, not in test environments. Chaos engineering proactively injects failures to discover weaknesses before they cause real incidents.
Chaos Engineering Principles
The Principles of Chaos Engineering (published by Netflix) define the methodology: (1) Start by defining “steady state” — the normal behavior of the system measured by business metrics (orders per minute, stream starts per second, API success rate). (2) Hypothesize that the steady state will continue in both the control group and the experimental group. (3) Introduce real-world events: server crashes, network partitions, disk failures, clock skew, dependency unavailability. (4) Try to disprove the hypothesis — look for differences in steady state between control and experiment. If the system maintains steady state despite the injected failure, confidence increases. If steady state is disrupted, you have found a weakness to fix before a real incident. The key insight: chaos experiments test the system behavior, not individual components. A service may handle a database failure correctly in isolation but fail when the retry storm from 100 instances overwhelms the database.
Netflix Chaos Monkey and the Simian Army
Chaos Monkey randomly terminates production EC2 instances during business hours. Purpose: force every team to build services that tolerate instance failure. If your service goes down because one instance was killed, it was not resilient enough. The Simian Army extended this concept: Latency Monkey injects artificial delays into RESTful client-server communication to simulate service degradation. Conformity Monkey finds instances that do not adhere to best practices and shuts them down. Chaos Gorilla simulates an entire AWS Availability Zone going down. Chaos Kong simulates an entire AWS Region failure. Netflix runs these tools continuously in production, not just during planned experiments. The result: Netflix services are designed from day one to tolerate failures at every level — instance, zone, and region. When AWS actually experiences an outage, Netflix is typically the last service to be affected and the first to recover.
Fault Injection Techniques
Common fault injection types: (1) Process killing — terminate a service instance to test automatic restart and failover. Tools: Chaos Monkey, kill -9 on Kubernetes pods (kubectl delete pod). (2) Network faults — inject latency, packet loss, or partition between services. Tools: tc (Linux traffic control) to add 500ms latency to a network interface, iptables to block traffic to a specific IP, Toxiproxy to simulate network conditions at the application level. (3) Disk faults — fill disk to 100% to test log rotation and alerting. Simulate slow I/O with dm-delay or fio. (4) CPU/memory stress — consume CPU or memory to test autoscaling and graceful degradation. Tools: stress-ng to generate CPU load, memory pressure cgroups. (5) Clock skew — shift the system clock forward or backward to test time-dependent logic (certificate validation, token expiry, scheduled jobs). (6) Dependency unavailability — block access to a database, cache, or external API to test fallback behavior and circuit breakers. Envoy proxy can inject faults at the service mesh level without modifying application code.
Game Days: Structured Chaos Experiments
A game day is a planned chaos experiment with a specific hypothesis, controlled blast radius, and a team ready to observe and respond. Game day structure: (1) Define the hypothesis: “If the Redis cache becomes unavailable, the checkout service will fall back to direct database queries and maintain sub-2-second response time.” (2) Define the blast radius: limit to one environment, one region, or one percentage of traffic. Start small. (3) Define the abort criteria: if error rate exceeds 5% or P99 exceeds 10 seconds, abort immediately. (4) Execute the experiment during business hours with the team watching dashboards. (5) Observe: did the system maintain steady state? What alerts fired? How did the on-call respond? (6) Document findings: what worked, what broke, what remediation is needed. Game days are also excellent for testing incident response processes — the team practices using runbooks, escalation paths, and communication channels under controlled stress.
Chaos Engineering Tools
Production-ready chaos engineering tools: (1) Litmus (CNCF) — Kubernetes-native chaos engineering platform. Injects pod kill, network loss, disk fill, and CPU stress via ChaosExperiment CRDs. Integrates with Argo Workflows for automated chaos pipelines. (2) Chaos Mesh (CNCF) — another Kubernetes-native tool with a web dashboard. Supports time chaos (clock skew), IO chaos (filesystem faults), and kernel chaos (kernel panic injection). (3) Gremlin — commercial chaos engineering platform with a SaaS control plane. Supports infrastructure (CPU, memory, disk, network), application (process kill, time travel), and state (shutdown, reboot) attacks across Kubernetes, VMs, and bare metal. (4) AWS Fault Injection Simulator (FIS) — AWS-native chaos tool that injects faults into EC2, ECS, EKS, and RDS. Supports terminating instances, throttling API calls, and simulating AZ failures. Integrated with CloudWatch for automatic rollback. (5) Toxiproxy (Shopify) — TCP proxy that simulates network conditions. Add latency, jitter, bandwidth limits, or connection resets to any TCP connection. Useful for testing application behavior under degraded network conditions in development and CI.
Building a Chaos Engineering Practice
Maturity progression: (1) Level 1 — manual experiments. Kill a pod in staging, observe what happens. No automation, no formal process. Good for building initial confidence. (2) Level 2 — planned game days in production. Monthly game days with specific hypotheses and documented results. Cross-team participation. (3) Level 3 — automated chaos in CI/CD. Chaos experiments run automatically as part of the deployment pipeline. A canary deployment includes an automated chaos test (kill 1 pod, inject 200ms latency) and verifies the service maintains its SLOs before promoting. (4) Level 4 — continuous chaos in production. Chaos Monkey runs continuously. The system is designed to tolerate ongoing failures without human intervention. This is the Netflix model. Start at Level 1 and progress as the team builds confidence and the system builds resilience. Do not jump to Level 4 — continuous production chaos on a system not designed for it causes real outages.