Don't Panic — Apache Iceberg

Apache Iceberg is what you reach for when a pile of Parquet files needs to behave like a real table. Not a warehouse you log into. Not the object store itself. A contract: engines agree on which files are in, which snapshot is current, and how a commit becomes visible without a prayer and a directory listing.

Before this sort of format, many lakes treated a table as folders plus a metastore that remembered partition paths. That worked until cloud buckets, huge partition counts, and concurrent writers showed up. Listings got expensive. Consistency got weird. Two jobs could disagree about what "the table" contained. Iceberg's answer is blunt: put the truth in a metadata tree, hang it off a catalog pointer, and swap that pointer atomically.

Three ideas carry almost everything else.

First, a snapshot is the whole table at one commit. Readers pick one and stick to it. They do not watch files appear mid-query like weather.

Second, hidden partitioning means you filter the columns you care about. Iceberg derives partition values with transforms such as day or bucket, then skips files that cannot match. When the layout needs to change, new data can use a new spec without forcing every old query to learn a new path convention.

Third, field IDs make schema changes boring in the good way. Add, drop, rename, and widen are metadata moves. They are not an invitation for a renamed column to inherit someone else's values because a name got recycled.

The surprise for many newcomers is operational, not conceptual. Iceberg will happily accept a streaming job that commits every minute. It will not quietly tidy the resulting small files, delete-file debt, and snapshot pile. Compaction, snapshot expiration, and orphan cleanup are part of running the system. Skip them and the demos still pass while planning latency and storage bills rehearse a mutiny.

Branches and tags are useful for write-audit-publish on a single table. They are not a multi-table git monorepo by themselves. If that is what you need, the catalog layer has more to say.

Read the Course intro when you want the architecture in full sentences. The Cheatsheet is the dense map of layers, write modes, and maintenance procedures. Practice and the exercise are for making a local spark-sql catalog do something irreversible-looking (in a warehouse directory you can delete). Field Notes is where the maintenance bill gets specific.

If you leave with only one sentence: Iceberg makes file collections into shared, versioned tables by committing metadata, not by hoping the bucket listing is telling the truth.

Don't Panic — Apache Iceberg

Relevant careers