Don't Panic — Systems Troubleshooting

Systems troubleshooting is a disciplined way to find why a system behaves differently from what you expect. Google's SRE book frames it as the hypothetico-deductive method applied to a running system: take observations and a model of how things should work, form hypotheses, test them, and keep what survives.

Before this was a method, a broken system was fixed by whoever built it, guessing at the most dramatic failure they could imagine. The method exists so that fixing a system does not depend on having built it, and so the cause is found before the responder runs out of time or evidence.

Three ideas hold the practice up. Triage before root cause: in a severe outage, stop the bleeding first — divert traffic, drop load, disable a subsystem — and preserve evidence while you do. Novice pilots are taught to fly the airplane first and troubleshoot second, and a responder who root-causes while the system dies is not helping users. Examine with metrics, logs, traces, and change history, none of which is a diagnosis on its own; the model of how the system should behave is what turns evidence into a hypothesis. And test the most likely and lowest-risk hypothesis first, one variable at a time, with the expected result and rollback stated before the change.

The one thing that surprises people: correlation is not causation. In a large system with many metrics, events correlate by coincidence, and two correlated events often share a common cause rather than one causing the other. A graph that lines up is an investigation target, not a conclusion.

The thing that catches every team: latching onto the cause of the last incident. It happened before, so it must be happening again — the SRE book names this as a common pitfall. "When you hear hoofbeats, think of horses, not zebras": prefer probable causes over dramatic ones, and let evidence rule the past out.

A subtle trap: the tools that give you evidence can change the system. Verbose logging worsens a latency problem, so the responder cannot tell whether the problem worsened on its own or because of the logging. Active tests have side effects, and a test that changes the system changes every later observation.

And the boundary on the whole effort: a root cause is often plural, not singular. Real systems often have several factors that are individually insufficient but jointly causative; a postmortem that names a single cause can satisfy a process while leaving the contributing conditions in place.

Read the Intro for the full hypothetico-deductive loop and the common pitfalls the SRE book names. The Cheatsheet holds the narrowing techniques — boundary walk, bisection, what/where/why — side by side. The Timeline shows how a practice that started with Toyota's Five Whys gained distributed tracing, the SRE book, OpenTelemetry, and eBPF — and why the method has stayed the same while the evidence got richer.