Chaos Engineering
Chaos engineering is the practice of intentionally injecting failures into a system to discover weaknesses before they cause outages. Teams form hypotheses about steady-state behavior, run controlled experiments, and use the results to improve resilience.
itPlatform engineering and SRE | OpenSkills.info
Intro
Chaos Engineering
Chaos engineering helps you answer a difficult reliability question with evidence: what happens when part of your system fails?
You run a controlled experiment against a working system. You define normal behavior, predict what should remain true, introduce a realistic disruption, and observe the result. The goal is not to cause damage. The goal is to find weak assumptions before an uncontrolled incident finds them for you.
Why this practice exists
A distributed system can fail even when each component appears healthy. A slow dependency can fill queues. A retry policy can multiply traffic. A database failover can expose stale connection handling. These behaviors emerge from interactions, so component tests alone cannot establish how the whole service responds.
Chaos engineering treats reliability as a claim you can challenge. A diagram may show redundant components. An experiment tests whether traffic actually moves to the redundant path within an acceptable time.
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://principlesofchaos.org/
Supports
- Chaos engineering definition and empirical systems focus
- Four-step experiment model using steady state, hypothesis, variables, and comparison
- Throughput, error rate, and latency as example steady-state measures
- Real-world variables, production realism, continuous automation, and minimized blast radius
- https://docs.aws.amazon.com/prescriptive-guidance/latest/chaos-engineering-on-aws/introduction.html
Supports
- Proactive resilience across people, process, and technology
- Distributed-system risks including dependency failures, retry storms, zone impairment, and resource exhaustion
- Controlled failures used to expose weaknesses before production disruption
- https://docs.aws.amazon.com/prescriptive-guidance/latest/chaos-engineering-on-aws/getting-started.html
Supports
- Observability, realistic scenarios, sponsorship, and remediation as prerequisites
- Business, server, customer, and operations metrics for experiment observation
- Failure scenarios across compute, APIs, networks, zones, regions, data, dependencies, traffic, resources, cascades, and deployments
- Historical incident replay and prioritization of remediation by business impact
- https://docs.aws.amazon.com/wellarchitected/latest/framework/rel_testing_resiliency_failure_injection_resiliency.html
Supports
- Regular experiments in or near production and their relationship to resilience testing
- Experiments as code, delivery-pipeline checks, and game-day inputs
- Faults for component loss, failover, resource pressure, latency, packet loss, and DNS failure
- Stop conditions as safety guardrails and passed experiments as regression tests
- https://sre.google/sre-book/testing-reliability/
Supports
- Testing as evidence that reduces uncertainty about future reliability
- Limits of nonrepeatable randomized techniques
- Capturing selected actions and turning observed failures into repeatable release tests
- Difference between evidence for a tested condition and proof of general reliability
- https://learn.microsoft.com/en-us/azure/chaos-studio/
Supports
- Managed resilience testing through workspaces and scenarios
- Custom targeted fault-injection experiments
- Scenario reports, fault catalogs, resource targeting, service limits, and known limitations