Don't Panic — Data Catalog and Discovery

A data catalog is an organized inventory of data assets (tables, files, streams, dashboards, services) described with metadata. It does not store your rows. It stores enough context that a stranger on a Tuesday can find the right asset and decide whether to trust it.

Before catalogs, people asked around. That works until the company has more tables than hallway conversations.

Three ideas carry almost everything else.

First, separate the dataset (the logical collection) from a distribution (a concrete way to get it) and from a catalog record (the registration of that description). Tools rename these; the jobs stay put.

Second, discovery needs four metadata layers: technical, business, operational, and relationship. Crawlers are good at the first. Owners, glossary terms, freshness, and lineage do the rest. Search that only knows column types returns structurally plausible assets nobody should ship a board metric from.

Third, metadata arrives by pull, push, and curation. Nightly crawls create coverage and staleness. Push events create fresher contracts when producers cooperate. Humans still have to write the sentence that says what the table is for.

The surprise: installing connectors is not the hard part. Empty descriptions, missing owners, and ghost entries for deleted tables are. Architecture essays about crawl-only portals versus event-sourced metadata platforms exist because trust dies when the inventory looks complete and feels wrong.

A catalog is also not your engine metastore, even when one cloud product plays both roles. The metastore helps compute read data. The catalog helps people discover assets across systems and attach ownership and policy context. Governance sets rules; lineage answers path questions; the catalog is where those meet the inventory.

Catalogs pay off when many teams produce and consume data across platforms and reuse matters. They are weak when a handful of well-known datasets already have clear owners, or when nobody will curate business metadata. In those cases the portal becomes a polite index of abandoned schemas.

If you read one more tab, use the intro for the full map, the cheatsheet for the layer and failure tables, Field Notes for the adoption traps, and Landscape when you need to compare products in this course's vocabulary.

Don't Panic — Data Catalog and Discovery

Relevant careers