Chaos Engineering
Chaos engineering is the practice of intentionally injecting failures into a system to discover weaknesses before they cause outages. Teams form hypotheses about steady-state behavior, run controlled experiments, and use the results to improve resilience.
itPlatform engineering and SRE | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic: Chaos Engineering
Chaos engineering is the alarming name for a disciplined test of a reliability claim. A working system receives a carefully chosen wobble, then gets watched to see whether important behavior stays inside an agreed boundary. It is not random breakage. That is sabotage with better branding.
The anchor is a steady state: acceptable behavior measured as successful orders, error rate, or response latency. Then comes a hypothesis, a prediction about what remains true during one disruption. Stop an application instance, for example, and successful checkout should remain within its boundary. Now there is a testable claim, rather than a diagram that hopes redundancy works.
The disruption is the chaos variable. It can be process loss, network delay, dependency error, resource exhaustion, failover, or a traffic surge. The surprising part is that breaking the component is the least interesting step. The value comes from comparing observations with the hypothesis, then fixing what the result exposed. A pass raises confidence for one condition, not for every possible future inconvenience. Systems are very fond of changing their minds.
Safety is part of the experiment. A blast radius sets its maximum scope. Abort conditions define when it stops. Begin with a small target, short duration, and low intensity, where impact is visible and recovery is possible. Production has real traffic and dependencies. It also has customers, which gives the word real an inconvenient amount of weight.
The Cheatsheet compresses the loop, scenario sources, safety gates, and result labels. Slides show the relationship between architecture, observability, experiments, incidents, and remediation. Field Notes covers the traps that make an experiment look controlled until it is not. The Reference tab supplies the authoritative path from principles to cloud and SRE guidance. Start with one customer-facing steady state, one credible fault, and one bounded run. Reliability becomes evidence, one carefully supervised inconvenience at a time.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://principlesofchaos.org/
Supports
- Chaos engineering definition and empirical systems focus
- Four-step experiment model using steady state, hypothesis, variables, and comparison
- Throughput, error rate, and latency as example steady-state measures
- Real-world variables, production realism, continuous automation, and minimized blast radius
- https://docs.aws.amazon.com/prescriptive-guidance/latest/chaos-engineering-on-aws/introduction.html
Supports
- Proactive resilience across people, process, and technology
- Distributed-system risks including dependency failures, retry storms, zone impairment, and resource exhaustion
- Controlled failures used to expose weaknesses before production disruption
- https://docs.aws.amazon.com/prescriptive-guidance/latest/chaos-engineering-on-aws/getting-started.html
Supports
- Observability, realistic scenarios, sponsorship, and remediation as prerequisites
- Business, server, customer, and operations metrics for experiment observation
- Failure scenarios across compute, APIs, networks, zones, regions, data, dependencies, traffic, resources, cascades, and deployments
- Historical incident replay and prioritization of remediation by business impact
- https://docs.aws.amazon.com/wellarchitected/latest/framework/rel_testing_resiliency_failure_injection_resiliency.html
Supports
- Regular experiments in or near production and their relationship to resilience testing
- Experiments as code, delivery-pipeline checks, and game-day inputs
- Faults for component loss, failover, resource pressure, latency, packet loss, and DNS failure
- Stop conditions as safety guardrails and passed experiments as regression tests
- https://sre.google/sre-book/testing-reliability/
Supports
- Testing as evidence that reduces uncertainty about future reliability
- Limits of nonrepeatable randomized techniques
- Capturing selected actions and turning observed failures into repeatable release tests
- Difference between evidence for a tested condition and proof of general reliability
- https://learn.microsoft.com/en-us/azure/chaos-studio/
Supports
- Managed resilience testing through workspaces and scenarios
- Custom targeted fault-injection experiments
- Scenario reports, fault catalogs, resource targeting, service limits, and known limitations
- https://netflix.github.io/chaosmonkey/
Supports
- Netflix created Chaos Monkey in 2010 to terminate instances and test resilience.
- https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116
Supports
- Netflix introduced the Simian Army in 2011.
- https://www.oreilly.com/content/chaos-engineering/
Supports
- The Principles were published in 2015 and describe control and experiment clusters.
- https://arxiv.org/abs/1702.05843
Supports
- A 2017 paper documented chaos engineering principles.
- https://www.cncf.io/blog/2022/01/11/litmuschaos-becomes-a-cncf-incubating-project/
Supports
- Litmus began in 2017, joined CNCF Sandbox in 2020, and reached incubation in 2022.
- https://www.cncf.io/blog/2022/02/16/chaos-mesh-moves-to-the-cncf-incubator/
Supports
- Chaos Mesh joined CNCF Sandbox in 2020 and incubation in 2022.
- https://aws.amazon.com/about-aws/whats-new/2021/03/aws-announces-service-aws-fault-injection-simulator/
Supports
- AWS announced general availability of Fault Injection Simulator in March 2021.
- https://learn.microsoft.com/en-us/shows/azure-friday/an-introduction-to-azure-chaos-studio
Supports
- Microsoft presented Azure Chaos Studio in December 2021.
- https://www.gremlin.com/community/tutorials/what-i-learned-running-the-chaos-lab-kafka-breaks
Supports
- Experiment duration must account for monitoring collection intervals.
- https://aws.amazon.com/fis/
Supports
- AWS Fault Injection Service product information.
- https://www.gremlin.com/
Supports
- Gremlin product information.
- https://www.steadybit.com/
Supports
- Steadybit product information.
- https://www.harness.io/products/chaos-engineering
Supports
- Harness Chaos Engineering product information.
- https://litmuschaos.io/
Supports
- LitmusChaos product information.
- https://chaos-mesh.org/
Supports
- Chaos Mesh product information.
- https://chaostoolkit.org/
Supports
- Chaos Toolkit product information.
- https://azure.microsoft.com/products/chaos-studio
Supports
- Azure Chaos Studio product information.
