Kubernetes Operations

Running applications on Kubernetes and running Kubernetes itself are different jobs, and this course is about the second one: the lifecycle work that keeps a cluster healthy over months and years. Upgrades, node maintenance, etcd care, capacity, certificates, and the discipline that ties them together. Even if a cloud provider manages your control plane, roughly half of this work remains yours — and knowing which half is itself operational knowledge.

The mental model: a cluster is cattle around a precious core. Nodes are replaceable by design — drain, upgrade or discard, replace. The control plane's state, concentrated in etcd, is the one thing that is genuinely irreplaceable. Operations is the practice of exploiting the replaceable parts' replaceability while guarding the core.

The rhythm: upgrades

Kubernetes releases minor versions roughly three times a year, and each receives patches for about 14 months. Staying within supported versions is not optional hygiene — it is the security patch supply. Upgrades follow a strict order dictated by the version skew policy: control plane components first (kube-apiserver leads; nothing may be newer than it), then node components (kubelet may lag the API server by up to three minor versions), then cluster add-ons. Minor versions cannot be skipped: 1.31 → 1.33 means passing through 1.32.

The operational craft is making this routine: upgrade one minor version at a time, control plane before nodes, with workloads that tolerate node churn (PodDisruptionBudgets, spread constraints — see the scheduling course). Managed services automate the mechanics but you still own the cadence, the add-on compatibility, and the deprecated-API cleanup that every release note lists.

Node lifecycle

Nodes join the cluster (self-registration or manual), report conditions (Ready, MemoryPressure, DiskPressure...), and are watched by the node controller, which taints unreachable nodes and eventually evicts their Pods. Planned maintenance uses the cordon/drain pattern: cordon marks a node unschedulable; drain evicts its Pods gracefully, honoring PodDisruptionBudgets, so controllers recreate them elsewhere. Then reboot, upgrade, or delete — the node is cattle. kubectl uncordon returns it to service.

This pattern — drain, mutate, return or replace — is the atom of all node operations: kernel patches, kubelet upgrades, instance-type changes, cluster scale-down. Immutable-infrastructure shops replace nodes rather than patching them; either way the workload-facing mechanics are identical.

The precious core: etcd

etcd holds every API object; its loss is the loss of the cluster's memory. Operations therefore treats it differently from everything else: run an odd number of members (quorum arithmetic — five members tolerate two failures), on fast disks (etcd is latency-sensitive), with TLS between peers and clients, and above all take and test backups. An etcd snapshot (etcdctl snapshot save, or filesystem-level backup) plus a tested restore procedure is the difference between an incident and a catastrophe. On managed control planes this is the provider's job — but backup of your cluster state (via tooling like Velero at the API level) remains a decision you own.

Certificates are the second control-plane care item on self-managed clusters: the PKI that secures component communication expires (kubeadm-issued certs after one year; kubeadm renews them on upgrade), and expired certificates are a classic self-inflicted outage.

Capacity, autoscaling, and cost

Cluster capacity management happens on two axes: the workload axis (requests/limits, HPA — covered in scheduling and workloads) and the node axis — adding and removing machines. The Cluster Autoscaler (and newer node-provisioning tools in the same family) grows node groups when Pods can't schedule and shrinks them when nodes sit underutilized, draining before removal. The operational responsibilities are the boundary conditions: sensible node group min/max, PDBs so scale-down can proceed safely, and periodic review of the requests-vs-usage gap that silently drives the bill.

The discipline

What separates calm clusters from eventful ones is rarely exotic tooling. It is: staying within the version support window; a tested etcd/state backup; PDBs on everything that matters; drains instead of node deletions; deprecated-API migration before, not after, the upgrade that removes them; and observability (previous course) good enough to notice trouble early. Kubernetes gives you the primitives — cordon, drain, skew policy, conditions; the discipline of applying them on a schedule is the actual job.

Limits and honest caveats

Operations is where Kubernetes' famous complexity concentrates. Self-managing control planes is a real engineering commitment — most teams are better served by managed offerings, spending their operations budget on the parts that remain theirs: node strategy, upgrade cadence, capacity, backup of application state, and add-on lifecycle. Multi-cluster architectures multiply every task in this course by N. And automation (GitOps pipelines, node auto-provisioning, operators) reduces toil but adds components that themselves need operating — adopt it for problems you actually have.