Don't Panic — Data Lineage

Data lineage is the record of how data moves and changes from a source to a destination. It exists because a dashboard number can have a surprisingly long family history: source datasets, transformation jobs, output datasets, metrics, and consumers. Without lineage, answering "where did that come from?" means walking through code, jobs, logs, and records one anxious clue at a time.

The useful picture is a graph, but the important part is the evidence behind it. Nodes name datasets, jobs, runs, and related things. Directed edges say what depended on what. Evidence adds the run, time, version, code, and owner that make an edge worth believing. A decorative graph is very good at decorating. It is less reliable at explaining a changed dashboard.

Two directions hold most of the subject together. Go upstream from an output toward its inputs and origins when an unexpected result needs tracing. Go downstream from an input toward derived assets and consumers before a change. The first route looks for the first unexpected state. The second finds who else may inherit it. Neither route proves correctness; logs, tests, source records, and domain knowledge still get the final vote.

The surprise is that names are part of the engineering. A dataset and a job need stable identities, commonly a namespace plus a name. Name the same asset differently and the graph splits it in two. Reuse one name for different assets and the graph invents a relationship. The chart has no way to feel embarrassed about either mistake.

The other surprise is that column-level lineage is not merely dataset lineage with smaller boxes. It can distinguish a copied field from a transformed or aggregated one, and it can record indirect influence through joins, filters, groups, sorts, windows, and conditions. That precision costs more to collect. Dynamic SQL, opaque services, and procedural code can keep part of the route offstage.

Start with the Intro for the full map: provenance concepts, capture sources, and the limits of the graph. Use the Slides for the relationships between dataset, job, run, and the two traversal directions. Keep the Cheatsheet nearby when scoping an incident trace or change review. The Field Notes are for the costly parts people discover after a graph has already become trusted. Then take one known workflow, define its identities, and test whether its path is actually present.

Don't Panic — Data Lineage

Relevant careers