Data Lineage

Data lineage records how data moves and changes from an origin to a destination. It connects source datasets, processing activities, outputs, and responsible systems or people. The result is a graph that can answer two practical questions: where did this data come from, and what depends on it?

The central mental model is a map plus evidence.

source dataset -> transformation job -> curated table -> metric -> dashboard
      node              node                node          node       node
          \____________ directed dependency edges ________________/

The map shows connected assets. The evidence describes the transformations, executions, times, versions, and owners behind those connections. A diagram without reliable evidence becomes stale. A stream of evidence without stable identities cannot form a useful graph.

Why data lineage exists

Modern data crosses databases, files, streams, transformation tools, schedulers, and reporting systems. A field on a dashboard may have passed through many jobs before a reader sees it. Local documentation rarely explains the entire path.

Lineage makes that path queryable. You can use it to:

trace a questionable result toward its upstream inputs;
identify downstream assets before changing a table or field;
discover which job produced a dataset;
connect an execution failure to affected outputs;
find owners and documentation attached to a path;
preserve evidence about how a result was produced.

Lineage does not prove that data is accurate. It records relationships and context. A fully traced calculation can still contain an incorrect rule.

A graph of entities and activities

The W3C PROV model gives provenance a general vocabulary. An entity is a physical, digital, conceptual, or other thing with fixed aspects. An activity occurs over time and uses or generates entities. An agent bears responsibility for an activity or entity.

For a data platform, you can map those ideas like this:

Provenance concept	Data example
Entity	File, table version, report, model, or published dataset
Activity	Query, ingestion task, transformation run, or export
Agent	Person, team, service account, or software system
Usage	A job reads an input dataset
Generation	A job creates an output dataset
Derivation	One dataset depends on another

OpenLineage applies a more operational model to data processing. Its core entities are datasets, jobs, and runs. A job describes recurring processing. A run identifies one execution of that job. Events connect the run to its input and output datasets. Facets attach details to the job, run, or datasets.

These models serve different scopes. W3C PROV is a general provenance interchange model. OpenLineage focuses on collecting lineage metadata from executing data systems.

Nodes need stable identity

A lineage graph fails when two names refer to one asset or one name refers to several assets. Stable identity is therefore a design requirement.

OpenLineage identifies a dataset with a namespace and name. It identifies a job the same way. A run uses an identifier maintained through that execution. The naming convention must stay consistent across emitters. Changing it later can split one logical asset into disconnected graph nodes.

Identity also needs the right granularity. A table name may identify a logical dataset, while a table version or partition represents a specific state. Choose identities that match the questions you need to answer. Impact analysis often starts with logical assets. Reproduction and audit work may require versions and execution times.

Dataset-level and column-level lineage

Dataset-level lineage records dependencies between whole datasets. It can show that a customer mart depends on an orders table and a customer table.

Column-level lineage records which input fields contribute to an output field. It can distinguish a copied identifier from a calculated amount. OpenLineage can also classify direct transformations and indirect uses such as joins, filters, grouping, sorting, windows, and conditions.

Column detail supports focused impact analysis, but it costs more to collect and maintain. Dynamic SQL, user-defined functions, procedural code, and opaque services can prevent complete inference. Start with the granularity your decisions require. Do not promise field-level certainty when the collector only observes dataset inputs and outputs.

Design lineage and runtime lineage

Lineage evidence can come from several places:

declarative model references and workflow definitions;
parsed SQL or code;
database query history;
scheduler and processing-engine integrations;
explicit lineage events emitted during execution;
manual relationships for opaque or external steps.

Design metadata describes what should depend on what. Runtime metadata describes what a particular execution used and produced. OpenLineage run events can record lifecycle transitions and attach input and output datasets to an execution.

Neither view is complete by default. A declared dependency may never run. A runtime collector may miss an external export. Comparing design and runtime views can reveal gaps, but the comparison is only as reliable as the captured metadata.

Capture, normalize, store, and query

A lineage system usually has four responsibilities:

Capture metadata near the system that knows the work.
Normalize identities and relationships into a common model.
Store nodes, edges, events, and facets with history.
Query upstream and downstream paths for a defined scope.

Capture close to execution when you need run evidence. A scheduler knows the task and run. A query engine knows the statement. A catalog knows asset descriptions and owners. No single integration necessarily sees the complete route from source to dashboard.

Normalization joins those views. It requires consistent namespaces, asset names, environments, and versions. Without that discipline, the store contains accurate fragments that do not connect.

Read the graph in both directions

An upstream query walks toward origins. Use it during incident investigation or when you need to understand how an output was produced.

A downstream query walks toward consumers. Use it for change impact analysis. Apache Atlas, for example, exposes lineage queries with input, output, or both directions and a configurable traversal depth.

Always define the starting asset, direction, depth, time, and environment. “Show all lineage” is rarely a useful request in a large system. A bounded question produces a reviewable answer.

A practical investigation pattern

Suppose a dashboard total changes unexpectedly.

Identify the exact metric or field and the affected time window.
Walk upstream from the dashboard to its serving dataset.
Find the run that produced the affected dataset state.
Inspect its job, inputs, code location, and attached metadata.
Continue upstream until you find the first unexpected change or failed assumption.
Walk downstream from that point to find every affected consumer.
Confirm the path against system evidence before taking corrective action.

Lineage narrows the search. Logs, data tests, source records, and domain knowledge establish the cause.

Ownership and operating quality

Treat lineage as a data product. Assign owners for integrations, naming rules, graph quality, and consumer support. Monitor whether expected emitters report, whether identities resolve, and whether critical paths reach their intended sources and consumers.

Useful quality checks include:

expected jobs produce recent run events;
known inputs and outputs connect to canonical asset identities;
critical datasets have upstream and downstream coverage;
duplicate nodes do not represent one logical asset;
retired jobs and assets have clear lifecycle treatment;
sensitive transformation details follow access controls.

Fresh events do not guarantee complete lineage. Coverage must be tested against known workflows and consumer questions.

Limits and poor fits

Lineage has several important limits:

A dependency edge does not prove transformation correctness.
Captured metadata can be incomplete, delayed, or misidentified.
A graph can expose names, code, schemas, or ownership details that require protection.
Dense graphs become hard to use without bounded queries and filtering.
Manual edges drift unless someone owns them.
Column-level inference may be uncertain when processing is opaque or dynamic.

Do not build a large lineage platform only to produce an attractive graph. Start with a decision such as impact analysis or incident tracing. Capture the minimum reliable evidence that supports it. Expand coverage when a real question requires more detail.

A practical learning path

Learn graph terms: node, edge, direction, path, and traversal depth.
Model datasets, jobs, runs, and ownership with stable identities.
Trace dataset-level paths upstream and downstream.
Add runtime events and distinguish job definitions from executions.
Introduce column-level lineage where field impact justifies the cost.
Measure coverage, identity quality, freshness, and unresolved edges.
Connect lineage to change review, incident response, governance, and reproducibility.
Study W3C PROV for broader provenance interchange and OpenLineage for operational collection.