Kubernetes Observability

A Kubernetes cluster is a distributed system running your distributed systems — so when something misbehaves, "look at the server" stops being an instruction anyone can follow. Kubernetes observability is about knowing where the platform's signals live — health probes, events, logs, metrics — and how they compose into an investigation path. The good news: Kubernetes is unusually self-describing, and most answers are one or two kubectl commands away if you know which layer to ask.

The mental model: observability in Kubernetes happens at two levels that must not be confused. The platform level — is the Pod scheduled, is the container restarting, is the node under pressure — and the application level — is your code doing the right thing. Kubernetes gives you the first natively and gives your applications the plumbing (probes, log capture, metrics APIs) to deliver the second. General observability theory is covered in the Observability Fundamentals course; this one is about where Kubernetes itself keeps the truth.

Health: probes tell the kubelet what "working" means

Kubernetes cannot guess what "healthy" means for your application, so you declare it with probes. A liveness probe answers "should this container be restarted?" — fail it and the kubelet kills and restarts the container. A readiness probe answers "should this Pod receive traffic?" — fail it and the Pod is removed from Service endpoints, no restart involved. A startup probe protects slow-starting applications by holding off the other probes until first success. Probes run as HTTP checks, TCP checks, gRPC checks, or in-container commands.

Probes are load-bearing: readiness gates rolling updates (a Deployment only proceeds when new Pods report ready) and traffic routing; liveness is the self-healing trigger. They are also the most commonly misconfigured observability feature — a liveness probe that checks a downstream dependency turns one slow database into a cluster-wide restart storm.

What happened: events and logs

Events are the platform's diary: scheduling decisions, image pulls, probe failures, evictions, all recorded as API objects and surfaced by kubectl describe and kubectl get events. They answer most "why is my Pod not running" questions directly, and they are retained only briefly (an hour by default) — ship them somewhere if you want history.

Logs follow one convention: containers write to stdout/stderr, the runtime captures the streams, and kubectl logs reads them — including --previous for the container that just crashed, which is routinely the single most useful debugging command in Kubernetes. Node-level rotation keeps files bounded; but node-level logs vanish with Pod eviction and node failure. Durable, searchable logging therefore means cluster-level logging — typically a node-level agent (a DaemonSet, e.g. Fluentd/Fluent Bit) shipping every container's streams to a backend. Kubernetes explicitly provides no native cluster-level log storage: that architecture decision is yours.

How much: the metrics pipelines

Resource metrics flow through two distinct pipelines that beginners regularly conflate. The resource metrics pipeline is minimal and built for automation: the kubelet (via cAdvisor) measures container CPU and memory, metrics-server aggregates cluster-wide, and the Metrics API serves exactly what kubectl top and the HorizontalPodAutoscaler consume. It is deliberately not a monitoring system — no history, no queries.

The full monitoring pipeline is the real observability stack: Kubernetes components expose rich Prometheus-format metrics on /metrics endpoints (API server, kubelet, scheduler, and friends), and the de facto standard is scraping them — plus kube-state-metrics (object-state metrics: Deployment replica counts, Pod phases) and your applications' own metrics — into Prometheus or a compatible backend. Kubernetes doesn't ship this pipeline; the ecosystem (Prometheus, Grafana, OpenTelemetry) standardized it.

The investigation path

Debugging on Kubernetes has a canonical order, and it saves hours: describe the Pod (events explain Pending, ImagePullBackOff, probe failures), logs with --previous (application errors, crash causes), exec into the container or use ephemeral debug containers (kubectl debug) when the image has no shell, then widen — node conditions, kubectl top, control plane component status. CrashLoopBackOff is a symptom, not a cause: the cause is in the previous container's logs or the events, nearly always.

Limits and honest caveats

Kubernetes hands you signals, not observability: no log storage, no metric history, no tracing, no dashboards, no alerting come built in — the platform assumes an ecosystem stack on top, and budgeting for one is part of running Kubernetes seriously. Events expire quickly. metrics-server data is instantaneous, not historical. Container logs die with their node. And every layer you add (mesh, operators, autoscalers) adds its own signals to learn. Start with probes done well, cluster-level logging, and a Prometheus-compatible metrics stack — that trio covers the first years of real operation.