Don't Panic - Kubernetes Observability

Kubernetes observability is about knowing where the platform's signals live (health probes, events, logs, metrics) and how they compose into an investigation path. A cluster is a distributed system running your distributed systems, so "look at the server" is rarely enough.

Work at two levels that must not be confused. The platform level asks whether a Pod is scheduled, a container is restarting, or a node is under pressure. The application level asks whether your code is doing the right thing. Kubernetes gives you the first natively and gives applications plumbing for the second.

Probes declare what healthy means. A liveness probe answers whether the kubelet should restart a container. A readiness probe answers whether the Pod should receive traffic. A startup probe protects slow starters by holding off the others until first success. Misconfigured liveness that checks a downstream dependency can turn one slow database into a restart storm.

Events record scheduling, image pulls, probe failures, and evictions. They answer many "why is my Pod not running" questions, and they expire quickly. Logs follow stdout/stderr: kubectl logs reads them, including --previous after a crash. Durable history needs cluster-level logging, usually a node agent shipping streams to a backend. Kubernetes does not store that history for you.

Resource metrics for kubectl top and HPA flow through metrics-server with no history. Rich monitoring scrapes Prometheus-format /metrics endpoints plus kube-state-metrics and application metrics into a compatible backend.

Investigate in order: describe (events), logs --previous, then widen to node conditions and control plane. CrashLoopBackOff is a symptom, not a cause.

Read the Intro for the two-level model. Use the Cheatsheet when you need the probe and command maps. Updates tracks Kubernetes releases that change these interfaces.