Don't Panic — Observability Fundamentals

The word is borrowed from control theory, where it describes how well a system's internal states can be inferred from its external outputs.

Ported into software it reduces to a test anyone can run this afternoon: can a question nobody anticipated be answered about a system already in production, without shipping code first? Yes means the system is observable. If the answer needs a new field and a deploy, what exists is monitoring — not an insult, merely a narrower thing.

Monitoring watches thresholds somebody chose in advance, and it is genuinely good at the failure modes that were predicted. Distributed systems mostly fail in ways nobody predicted, across services and queues and bought-in components nobody operates directly, in configurations that refuse to reproduce on a laptop.

The extension exists so a novel problem can still be investigated afterwards, out of signals gathered before anyone knew they would matter. Which is why the property is bought at instrumentation time rather than at purchase time: no dashboard product confers it.

The signals divide the work. Metrics are numbers aggregated over time — cheap, quick to query, volume unaffected by traffic, the right backbone for alerting, and they have thrown the detail away.

Logs are timestamped records of individual events: rich, universally available, unpleasant at scale.

Traces follow one request across process and service boundaries, assembled from spans — the individual units of work — and show where the time went.

None is sufficient alone. Metrics say what is wrong at predictable cost; logs and traces say why, at a far higher cost per event. So the grown-up move is correlation: shared labels, trace identifiers written into log lines, sample requests attached to an aggregated metric so a spike leads back to one real case.

Beside that sits the vocabulary used in meetings, and its order is the point. Pick a number reflecting something a user actually experiences and that is the indicator, the SLI; commit to a target for it and you have an objective, the SLO; let somebody sell a contract on that target, with penalties, and it becomes an agreement, the SLA. Whatever the objective leaves spare is the error budget.

Latency, traffic, errors and saturation are the four golden signals, the least worth measuring on anything user-facing.

Now the tension that shapes the entire subject. The fields that answer new questions are exactly the ones the cheapest storage punishes. Novel incidents get resolved by high-cardinality dimensions — ones with very many distinct values, such as a tenant, a build identifier, a feature flag — and every combination is another series to pay for. Choose which few are worth carrying, and put them on traces or events rather than metrics.

Two more worth arriving with. Alert on what users feel rather than on causes: cause-based alerts multiply with every component added, most fire when nobody was harmed, and a page deserving only a robotic response should not be a page.

And the telemetry bill scales with architecture rather than traffic — splitting one service into eight multiplies spans per request and adds a dimension to every metric at identical load.

Slides if you want the shape fast, Cheatsheet for the signal comparison. Then Field Notes, for the one uncomfortable number nothing you buy can improve.

Don't Panic — Observability Fundamentals

Relevant careers