Distributed Tracing

A request can cross an API gateway, several services, a queue, and a database. A dashboard may tell you the request was slow. A distributed trace shows the path it took and where the time went.

Distributed tracing records one request as a trace. Each unit of work becomes a span. Parent and child relationships connect those spans into a causal story across process and network boundaries.

Why tracing exists

Logs describe events inside individual components. Metrics summarize behavior across many operations. Neither signal automatically preserves one request's path through a distributed system.

Tracing adds that request-level structure. You can use it to:

• locate the service or dependency that added latency;
• see which branch returned an error;
• understand retries, fan-out, and asynchronous work;
• correlate a trace with logs that carry its trace ID;
• compare the observed path with the architecture you intended.

Tracing is especially useful when a problem crosses ownership or service boundaries. It complements metrics and logs. It does not replace them.

The mental model

Think of a trace as a tree of timed operations.

checkout request                         root span
├── validate cart                       child span
├── reserve inventory                   child span
│   └── database query                  child span
└── authorize payment                   child span

The root span represents the overall operation. Child spans represent work performed within it. Each span normally records a name, start and end times, status, attributes, events, and a span context.

A trace ID identifies the complete trace. A span ID identifies one span. A parent span ID establishes the immediate relationship between two spans.

Context propagation makes it distributed

The next service must receive trace context before it can attach its work to the same trace. Instrumentation injects context into a carrier, such as HTTP headers. Receiving instrumentation extracts it and creates a child span.

OpenTelemetry uses W3C Trace Context by default. Its traceparent header carries a version, trace ID, parent ID, and trace flags. The optional tracestate header carries vendor-specific trace information.

Broken propagation produces separate trace fragments. This is one of the first things to check when spans exist but do not form an end-to-end trace.

What a useful span contains

Use stable, low-cardinality span names that describe a class of operations. Put request-specific values in attributes instead of the name. For example, GET /users/{id} is a useful operation name. A name containing a specific user ID creates too many distinct groups.

Attributes add searchable metadata. Events record notable points in time within a span. Status describes the operation's outcome. Links connect spans when a strict parent-child relationship does not fit, such as some asynchronous workflows.

Span data can contain sensitive values. Treat attributes, events, baggage, and propagated headers as data with security and privacy consequences. W3C Trace Context forbids personally identifiable or sensitive information in traceparent and tracestate.

Collection path

The typical path has four stages:

1. Instrumentation creates spans and propagates context.
2. A tracing SDK applies sampling and processes completed spans.
3. An exporter sends spans directly or through an OpenTelemetry Collector.
4. A trace backend stores, queries, and visualizes the trace.

OpenTelemetry separates instrumentation from a specific backend. Its exporters can send data to the Collector or another consumer. This separation lets teams change collection and storage choices without rewriting every instrumentation point.

Sampling and cost

Recording every trace may create unacceptable processing, network, and storage costs. Sampling selects which traces are exported.

Head sampling decides near the start of a trace. It is fast, but it cannot know the final outcome. Collector-side or tail-based strategies can consider later span data, but they require infrastructure that can assemble enough of the trace before deciding.

A sampling policy trades completeness for cost. Validate it against the questions you need to answer. Rare errors and high-latency requests can disappear when a policy ignores outcomes.

Limits

Tracing cannot explain work that was never instrumented. Missing context propagation breaks causal relationships. Poor span names and inconsistent attributes make queries unreliable. Excessive attributes increase cost and may expose sensitive data.

A trace also shows only the sampled request. Use metrics to understand population-wide rates and distributions. Use logs for detailed component events. Correlate all three signals when possible.

A practical learning path

Start by reading traces and identifying roots, parents, children, timing, and errors. Then learn context propagation and instrument one synchronous request path. Add consistent semantic attributes and log correlation. Next, operate the SDK-to-Collector export path and measure dropped spans. Finish with sampling policy, asynchronous traces, and privacy review.