Mostly harmless, conspicuously useful

The Hitchhiker's Guide to Becoming a Site Reliability Engineer

A site reliability engineer applies software engineering to the problem of keeping a production service alive, which is a perfectly reasonable ambition until you remember that "alive" is defined by a number on a dashboard, the number is defined by an objective the business once agreed to in a meeting whose minutes have since been lost, and the service has been arguing with the definition ever since. You learn to translate "it works" into service level indicators, objectives, error budgets, runbooks, capacity plans, and alerts that name their cause before the customer names it for you. SRE is not operations with a newer job title; it is the realisation that reliability is a feature with a budget, that toil is a finite resource best spent oneliminating itself, and that a pager is a small electronic device whose principal product is interrupted sleep. This guide travels from reading one service's dashboard to setting organisation-wide reliability strategy, with practical stops at observability, incident command, error budgets, automation, capacity, and the recurring discovery that an alert which fires every Monday is not an alert, it is a calendar invite the team has learned to ignore. The grand objective is dependable service measured against an honest target; the daily evidence is usually a dashboard that reconciles, an incident that closed itself in under the objective, and a pager that has, for once, nothing further to add.

Things to recognise

• A green dashboard proves the configured SLI is within its objective; it does not prove the SLI measures what the customer experiences, the window is honest, or the average is not concealing a long outage in a short month.
• Large gaps between the numerator and denominator definitions suggest the SLI is measuring a neighbouring question, and the dashboard may confidently report "available" while the customer reports nothing of the kind.
• Confirm the service, window, and SLI definition before declaring the objective met, because a correct number against the wrong denominator remains impressively incorrect and surprisingly durable.

Take the useful exit

Level 1 · Novice

Read the dashboard before asking the service how it feels

You inspect service dashboards, SLOs, alert definitions, and recent incident notes without changing them, learning how a perfectly polite metric can tour every chart, cross every threshold, and report exactly what was measured rather than what was meant.

You begin with read-only artifacts: a service dashboard, its service level indicator definitions, the SLO document, recent incident timelines, an alerting rules file, and an on-call schedule expressing recent opinions. A service level indicator is the specific quantity being measured — availability, latency, freshness, success rate; an objective is the target value the service agrees to meet over a window; an error budget is the slack that objective leaves for things to go wrong, which is also the slack the service will spend with the enthusiasm of a clerk drawing down a generous expense account. You review these with developers, on-call operators, product owners, and an experienced engineer so everyone can trace how "reliable" is measured before anyone declares the service is or isn't.

Suppose a service's availability dashboard shows 99.97% against a 99.9% objective. In a read-only view, you inspect which requests the SLI counts, the window it averages over, the denominator (all requests or only those that reached the service), and whether the 0.03% gap is concentrated in one minute or smeared across the month. You record the SLI definition, the window, the numerator and denominator, and whether the dashboard's green badge celebrates an average that hides a forty-minute outage in an otherwise uneventful Tuesday. That evidence lets a developer fix a misleading indicator or escalate a real regression; one brisk dashboard glance is an anecdote with good posture, not a diagnosis.

Words from the spaceship manual, translated

Service level indicator: The specific quantity measured to judge a service — availability, latency, success rate, freshness. It is what the dashboard actually shows, which is not always what the team thinks it is arguing about.
Service level objective: The target value a service agrees to meet over a defined window, such as 99.9% availability over 28 days. It turns "fairly reliable" into a number a stopwatch can cross-examine and an alert can argue with.
Error budget: The slack an objective leaves for failure: at 99.9% availability over 30 days, the service may be down for roughly 43 minutes before the budget is spent. It is the allowance the service treats as a target and the business treats as a surprise.
Alert: A notification that something requires a human, defined by a rule over a metric. A good alert names its cause, its owner, and its response; a bad alert is a calendar event with a siren attached, which the team will learn to ignore precisely when it begins to matter.