Mostly harmless, conspicuously useful

The Hitchhiker's Guide to Becoming a Systems Administrator

A systems administrator operates servers, operating systems, identities, storage, and the core services on which everyone else's confidence depends, which is a perfectly reasonable profession built atop a shell script that has worked for fifteen years and must not be touched. You learn to translate "the server is slow" into a question about processes, memory, disk, network, a package manager's feelings, and a cron job that runs as root because someone in 2008 decided it should and no one since has felt sufficiently energetic to disagree. The operating system is welcoming on the way up and commemorative on the way down, and an uptime number displayed with pride is a vanity metric that measures how long a host has avoided both rebooting and security patches. The package manager updates packages, holds opinions about dependencies, and is prepared to upgrade one library into incompatibility with another while remaining politely optimistic about the outcome. This guide travels from reading a process table without flinching to designing organisation-wide automation and recovery, with practical stops at storage, identity, virtualization, backup, automation, troubleshooting, and the recurring discovery that a host marked "healthy" is merely healthy enough to bill and not yet healthy enough to serve customers. The grand objective is dependable service at a defensible level of effort; the daily evidence is usually a process that exits on its own, an alert that names its cause, and a monthly patch window that fails to surprise anyone except the application team, who have learned not to be surprised by their lack of surprise.

Things to recognise

• A filesystem reported as full may be full of inodes rather than bytes, so a disk with free space that refuses new files is not haunted; it has run out of names for the files it already has.
• Processes do not announce their exit, so a service that "stopped" may have been killed by the out-of-memory handler, by a parent, or by a signal from a script whose name nobody remembers; check the journal before restarting, or you will restart the cause too.
• Confirm the host, service, filesystem, and write capability before changing configuration, because a correct edit on the wrong host remains impressively persistent and surprisingly difficult to undo.

Take the useful exit

Level 1 · Novice

Read the process table before the kernel forms an opinion about you

You inspect processes, memory, filesystems, logs, and configuration without changing services, learning how a perfectly polite kernel can run out of memory and begin assassinating processes before anyone has finished the sentence describing what ought to run.

You begin with read-only artifacts: process tables, memory and swap usage, filesystem layouts, boot records, service journals, package inventories, and a `/etc` directory expressing recent opinions about how the host should behave. An operating system is the program that manages processes, memory, files, and devices so other programs can pretend the hardware is someone else's problem. A process is a running program with a number, a parent, and a set of opinions about memory it will defend with signals. You review these with developers, security staff, and an experienced administrator so everyone can trace how processes, storage, and services connect before anyone runs a command whose confirmation dialog is also a configuration change.

Suppose a host's web service becomes unresponsive. In a lab machine with no production load, you read the process list, confirm the service is running, inspect memory and swap, read the journal, and note that a filesystem reported as full is a polite refusal rather than a software bug. You record the inode count, the one log file that has grown without supervision since 2019, and the package version that is older than the policy that demands it be newer. One tidy process table is an anecdote with good posture, not an architecture; but it prevents the team from commissioning a second server for a problem that was, in the end, a log file.

Words from the spaceship manual, translated

Process: A running program with an identifier, a parent, and an allocation of memory it regards as its own. Processes are plentiful, chatty, and unembarrassed about consuming resources until the kernel intervenes with prejudice.
Filesystem: The structure that organises files on storage so they can be found, mounted, and, on a bad day, repaired. A filesystem that is full is a host that has run out of opinions and begun refusing new ones, often in the middle of a transaction.
Service: A program managed by the init system so it starts, stops, and restarts according to declared intent. A service that restarts alone is technically alive in the manner of an empty shop with the lights on and a status that says so.
Journal: The collected log output a modern init system keeps for each service, so that operators can ask a running host why it behaved a certain way without first finding the log file it was using as a diary.