Don't Panic — Grafana Loki

Loki is a log aggregation system with a strict budget. It keeps every log your infrastructure produces, makes them searchable, and pays for it all by refusing to index the words. Only the labels get indexed; the log lines themselves live, compressed, in the same kind of object storage that holds your backups. That is the entire trick, and the entire product.

The problem it replaced was the full-text index. Systems like Elasticsearch index every field of every line, which is magnificent and expensive: an index roughly as large as the logs, on hardware you provision, patch, and mourn. Loki's designers looked at what people actually type into log search boxes, usually "show me this app's errors in production", and noticed those questions need a handful of labels, not every word. So Loki keeps a small index as a table of contents and greps the matching compressed chunks at query time.

Three ideas do the heavy lifting.

Streams. Every log line gets labels at the moment it is shipped: environment, namespace, app name. Every unique combination of label values is its own stream, stored separately. This is the most consequential design in Loki, and it is also the trap: give in to the temptation to label requests by client IP or status code, and every value becomes its own stream, your index balloons, and your queries crawl. Bounded values only. Everything high-cardinality, trace IDs, pod names, user IDs, goes into structured metadata, which is stored but never indexed.

The pipeline. An agent, Grafana Alloy these days, collects logs and pushes them over HTTP. Inside Loki, a distributor validates and rate-limits, ingesters hold chunks in memory with a write-ahead log for safety, and full chunks flush to object storage. Reads split across queriers that fetch from memory for recent logs and storage for old ones. Each piece scales on its own, or the whole thing runs as one binary on a Raspberry Pi. The same binary, in fact; the difference is a startup flag.

LogQL. The query language is Prometheus's, applied to logs. A selector in braces, then filters: contains this, exclude that, parse the JSON, compare the parsed field. And the trick that makes it feel like a superpower: wrap a log query in count_over_time or rate, and your logs become metrics. Those metrics feed dashboards, and the ruler evaluates them continuously and fires alerts. Logs to metrics to pages, without leaving the log system.

What will surprise you? Two things, and Loki admits both openly. Retention is off by default: without explicitly enabling the compactor, Loki keeps everything forever, which is either a feature or a finance meeting, depending on when you notice. And quiet periods return "no data" rather than zero, a small cruelty that breaks alert rules until you learn the or on() vector(0) incantation.

What Loki is not: a free-text search engine. If your work means searching every word of every log over years, security forensics in particular, Elasticsearch-class systems remain the right tool. Loki's bet is that most log questions start from "which app, which environment", and it wins that bet on price every time.

If you read one more thing, make it the Cheatsheet: the LogQL grammar and the label rules in one screen. The Landscape tab shows the neighbors, Fluent Bit, Elasticsearch, the whole Grafana stack. And the Reference tab starts, pointedly, with the labels page, because in Loki, labels are the whole ballgame.

Don't Panic — Grafana Loki

Relevant careers