Observability Fundamentals

Observability is the property of a system that lets you understand it from the outside: you can ask questions about its state and behavior without knowing its inner workings, and without shipping new code first. The term comes from control theory, where observability is "a measure of how well internal states of a system can be inferred from knowledge of its external outputs." In practice, it is what lets you troubleshoot problems you never anticipated — the unknown unknowns — and answer the question "why is this happening?" rather than just "is it happening?"

The useful mental model: your system continuously emits telemetry — data about its own behavior — and observability is your ability to turn that output into answers. A system does not become observable because you bought a dashboard product. It becomes observable because it was instrumented to emit the right signals, and an application is properly instrumented when engineers don't need to add more instrumentation to troubleshoot an issue, because the information they need is already there.

Why observability exists

Cloud computing, microservices, and distributed systems moved applications onto infrastructure their operators no longer fully control. A request may cross dozens of services, queues, and managed cloud components before failing — and the failure often cannot be reproduced locally. Traditional monitoring, which watches predefined thresholds on predefined metrics, catches the known unknowns: failure modes someone predicted in advance. Distributed systems routinely fail in ways nobody predicted. Observability extends monitoring so that novel problems can still be investigated after they happen, using signals collected before anyone knew they would matter.

This is also an organizational shift. Instrumentation used to be an afterthought bolted on by operators; the modern approach makes it a developer concern, designed in from the start — what the CNCF calls open-box monitoring, as opposed to close-box monitoring that probes a system purely from outside.

The primary signals

You will hear about the "three pillars of observability." The CNCF prefers primary signals, because the list is neither fixed nor always needed all at once:

• Metrics — numeric measurements aggregated over time: request rates, error rates, CPU utilization. Cheap to store, fast to query, and their volume does not grow with traffic, which makes them the backbone of alerting, dashboards, and trend analysis. The trade-off: aggregation discards detail.
• Logs — timestamped text records of discrete events, from application errors to security audit trails. Rich in detail and available almost everywhere, but freeform text is hard to analyze at scale, and logs alone usually lack the context of where a request came from.
• Traces — records of a single request's journey through a distributed system, built from spans (units of work) linked by context propagation. Traces show where time was spent and which component failed, across process and service boundaries.
• Profiles and crash dumps — emerging signals. Continuous profiling shows which code consumes CPU or memory; core dumps capture the state of a crashed process.

No single signal is enough, because each is specialized. Metrics tell you what is wrong at scale for a predictable cost; logs and traces tell you why, at a much higher per-event cost. Mature setups correlate signals — shared labels across signals, trace IDs attached to log lines, and exemplars linking a spiking metric to a sample trace — so an investigation can jump from an alert to the exact failing request.

The vocabulary of reliability

Observability data underpins how organizations define and defend reliability:

• An SLI (service level indicator) is a quantitative measure of service behavior from the user's perspective, such as page load time or error rate.
• An SLO (service level objective) is the target for an SLI — the way reliability is communicated to the rest of the organization.
• An SLA (service level agreement) is a business contract with consequences for missing the objective.
• The error budget is the tolerated failure allowance implied by an SLO: 100% minus the SLO over a time window.

Google's SRE practice adds the four golden signals — latency, traffic, errors, and saturation — as the minimum worth measuring for any user-facing system, and a discipline for alerting: page a human only for urgent, actionable, user-visible symptoms; route everything else to tickets and dashboards.

Who uses it, and for what

Site reliability engineers, DevOps engineers, sysadmins, and software developers all consume observability data, for purposes beyond firefighting: analyzing long-term trends and capacity, comparing releases and experiments, alerting, building dashboards, and retrospective debugging. Product teams use the same data to understand how features are actually used.

Limits and failure modes

Observability is a game of trade-offs, and the common failures are predictable. Collecting everything "because you never know" builds an expensive data lake, not an observable system — you need an objective before you choose outputs. Metrics with too many unique label values cause cardinality explosions that break or bankrupt metric storage. Noisy, unactionable alerts cause alert fatigue that trains people to ignore real pages. And logs are not guaranteed delivery, so genuinely important data should never live only in logs — nor should personally identifiable information live in them at all.

The pragmatic starting path, per the CNCF: begin with the signal you already have — metrics in cloud-native environments, logs in more traditional ones — get it into good shape, then branch out.