Don't Panic — Distributed Tracing

Distributed tracing is the record of one request as it wanders through a distributed system, collecting timed pieces of work called spans. The request may begin at a gateway, visit several services, leave a note at a queue, and ask a database an awkward question. Metrics can report that something is slow. Logs can report what one component said. A trace is the evidence that lets those statements refer to the same journey.

The important shape is a tree. A root span represents the overall operation, and child spans represent work caused by it. Each span has a trace ID, a span ID, timing, status, attributes, events, and sometimes links. This sounds like bookkeeping because it is bookkeeping. In a system with many services, accurate bookkeeping is what keeps a timeout from becoming a small archaeological expedition.

The bit that makes the tree cross service boundaries is context propagation. Sending instrumentation puts trace context into a carrier, such as request headers. Receiving instrumentation extracts it and creates a child span. OpenTelemetry uses W3C Trace Context by default, including the traceparent header. If injection or extraction fails, the trace does not become slightly less tidy; it becomes disconnected fragments that cannot show the end-to-end path.

A useful trace also needs useful span design. Stable operation names group similar work, while request-specific values belong in attributes. Attributes and events are valuable until they contain secrets, personal data, or unbounded values. Sampling then decides which trace data is exported. An early decision is cheaper but cannot know the final error or latency. A later decision knows more but needs more collection infrastructure. There is no magic setting, only a policy that preserves the incidents worth investigating.

Start with the Intro for the complete model and collection path. Use the Slides for the relationships between traces, metrics, logs, propagation, and sampling. Keep the Cheatsheet open while reading an unfamiliar waterfall. Then use the practice reference and exercise to prove one request remains connected across a real boundary. A long span is evidence, not a verdict, which is an excellent habit to acquire before the graph starts looking persuasive.

Don't Panic — Distributed Tracing

Relevant careers