Data Lakes and Lakehouses

A data lake is an architectural approach for keeping varied data in a shared repository. You can retain structured tables, events, documents, images, and other data before every use is known.

A lakehouse adds a table-management layer over lake storage. That layer gives files database-like behavior, including transactions, versioned table state, schema controls, and query-planning metadata.

The central mental model is files below, tables above.

producers -> ingestion -> object storage and data files
                              |
                              v
                    table metadata and catalog
                              |
                              v
                SQL, streaming, notebooks, ML, and BI

Storage holds objects. A file format organizes values inside each object. A table format records which files form a table at a given point. A catalog helps engines and people find that table. Compute engines read the metadata and files to perform work.

These layers are separate choices. Calling object storage a lakehouse does not make it one. The table-management layer is the defining addition.

Why data lakes exist

Operational databases optimize application transactions. Data warehouses organize curated analytical data for reporting and business intelligence. Neither role covers every data shape or every analytical workload.

A data lake provides a common storage foundation. Producers can land source data, then different engines can transform or analyze it. Cloud designs often separate storage from compute, so each can scale and change independently.

This flexibility supports several needs:

retain source data before all downstream questions are known;
combine structured, semi-structured, and unstructured data;
use different processing engines over shared storage;
prepare data for SQL analysis, data science, and machine learning;
keep raw, transformed, and curated data under one governance model.

Flexibility does not remove engineering work. A usable lake still needs metadata, ownership, access control, quality controls, lifecycle rules, and reliable pipelines.

Why lakehouses exist

An unmanaged collection of files is difficult to update as one logical table. A table may span thousands of objects. Readers need a consistent answer to a basic question: which files belong to the table now?

The lakehouse design answers that question with metadata. A table format tracks table state separately from the contents of each data file. A commit creates a new table state. Readers select a valid state instead of guessing from a directory listing.

This design brings several management capabilities to lake storage:

atomic table commits;
consistent reads during concurrent work;
snapshots and time travel;
schema evolution;
partition evolution;
metadata-based file pruning;
updates, deletes, and merges where the format and engine support them.

The original lakehouse paper describes a metadata, caching, and indexing layer over data-lake files. It aims to serve warehouse-style SQL and direct data-science access without maintaining a separate copy for every workload.

The idea is an architecture, not one product. Apache Iceberg and Delta Lake are examples of table formats that implement parts of this layer.

The five-layer stack

Keep five layers distinct when you design or troubleshoot the platform.

Layer	Main question	Examples
Storage	Where do objects live?	Object storage or a distributed file system
File format	How are values encoded inside a file?	Parquet, ORC, JSON, CSV
Table format	Which files form the table, and what is its current state?	Apache Iceberg or Delta Lake
Catalog and governance	How do users find, own, and access tables?	A metastore, catalog, policy service, and lineage system
Compute	Which engine reads, writes, transforms, or serves the data?	SQL engines, stream processors, notebooks, and ML tools

Apache Parquet is a column-oriented file format. It organizes a file into row groups, column chunks, and pages. Readers can select required columns and use metadata to avoid some irrelevant data.

Parquet does not define the state of a multi-file table. That is the table format's job. Confusing these layers leads to fragile designs, such as treating a folder of Parquet files as a transactional table.

Snapshots, transactions, and concurrency

A table snapshot represents one valid state of the table. In Apache Iceberg, table metadata points to snapshots. Snapshots point through manifest metadata to data files.

A writer prepares new data and metadata, then attempts an atomic change to the current table metadata. Optimistic concurrency lets writers proceed without holding one long global lock. A conflicting commit must be validated and retried or rejected.

Readers use a committed snapshot. They do not observe a half-finished set of file changes. Older snapshots can support time travel while their metadata and files remain retained.

Time travel is useful for audits, reproducible analysis, investigation, and rollback. It is not free backup. Retention policies may expire old snapshots and remove files that no valid snapshot references.

Schema and partition evolution

Lake data changes. Producers add fields. Query patterns shift. Data volume grows.

Schema evolution changes the logical columns of a table. Apache Iceberg assigns stable identifiers to columns. This lets it distinguish a renamed column from an unrelated column that later reuses the old name.

Partitioning groups related rows during writes so engines can skip irrelevant files. Traditional folder-based partitioning often exposes the physical layout in queries. Iceberg uses hidden partitioning. A query filters a source column, and the table format derives the relevant partition values.

Partition evolution changes the layout for new data without eagerly rewriting old data. Metadata keeps the different layouts separate, and scan planning handles both.

Evolution still needs compatibility policy. A technically supported change can break an engine, dashboard, or data contract that assumes the old shape.

Zones are lifecycle boundaries

Many lakes separate data into raw, transformed, and curated zones. The names vary, but the responsibilities are useful.

Zone	Purpose	Typical controls
Raw	Preserve source-aligned data for replay and investigation	Restricted writes, retention, source metadata
Transformed	Standardize types, structure, and reusable logic	Quality checks, deduplication, schema rules
Curated	Present trusted data for defined consumers	Ownership, service expectations, semantic definitions

A zone is not a quality guarantee. Moving a file into a curated path does not prove its accuracy. Each boundary needs explicit entry criteria, ownership, and evidence.

Do not create copies without a purpose. A new zone is justified when it creates a meaningful control, interface, retention rule, or consumer contract.

Catalog, governance, and security

Data becomes usable when people and engines can discover its meaning and access it safely.

A technical catalog records locations, schemas, partitions, and update metadata. Business metadata adds descriptions, owners, sensitivity, and intended use. Governance connects that metadata to policies and accountability.

Access control should cover both metadata and underlying data. A user who cannot discover a table may still reach its objects through another path unless both layers are governed. Central policy systems can apply permissions at database, table, column, row, or cell level, depending on the platform.

Plan these controls before broad ingestion:

ownership and stewardship;
classification and sensitive-data handling;
least-privilege access;
encryption and key management;
audit records;
retention and deletion;
lineage and approved uses;
quality and freshness expectations.

A lake without these controls can become easy to fill and hard to trust.

Performance and maintenance

Open storage does not guarantee fast queries. Performance depends on file layout, metadata, engine behavior, and workload shape.

Columnar formats help analytical scans because engines can read selected columns. Partitioning and file statistics help skip irrelevant files. Table metadata can prune manifests and data files before execution.

Writes can produce many small files. Each file adds metadata and open cost. Compaction rewrites smaller files into a more useful layout while preserving the live records.

Snapshots and failed writes can also leave metadata or unreferenced files. A production table needs scheduled maintenance for compaction, snapshot expiration, and orphan-file cleanup. Retention must match audit, recovery, and regulatory needs before deletion runs.

Measure the whole path. A fast engine cannot compensate for uncontrolled file counts, poor partition choices, missing statistics, or an overloaded catalog.

Data lake, warehouse, or lakehouse?

Treat the choice as a workload decision, not a contest.

Need	Data lake	Data warehouse	Lakehouse
Retain varied source data	Strong fit	Often secondary	Strong fit
Managed relational tables	Requires added services	Native focus	Added through table metadata
Direct file access	Core property	Usually abstracted	Preserved by design
Cross-engine access	Possible, with compatibility work	Product-dependent	A goal, still requires testing
Transactional table changes	Not inherent	Native focus	Supplied by table format and engine
Operational simplicity	Requires assembling controls	Often integrated	Requires operating storage, metadata, and compute layers

A warehouse remains a good choice for governed SQL reporting with predictable relational workloads. A data lake remains useful for archival, exploration, and varied data that does not need transactional tables. A lakehouse fits when several analytical workloads need governed tables over shared lake storage.

Hybrid architectures are normal. You may keep source data and ML features in a lakehouse while serving a tightly managed reporting workload from a warehouse. Every extra copy needs freshness, reconciliation, cost, and ownership controls.

Common use cases

Data lakes and lakehouses support work such as:

retaining events and source extracts for replay;
preparing large analytical datasets;
combining batch and streaming inputs;
serving SQL analysis from shared object storage;
building reusable data-science and machine-learning datasets;
sharing governed tables across teams and engines;
keeping historical snapshots for reproducible analysis.

Limits and poor fits

A lakehouse is not an operational transaction database. Table formats optimize analytical data, not low-latency record-by-record application transactions.

It also does not remove data modeling. Raw flexibility postpones some decisions, but consumers still need stable meaning, grain, keys, and quality rules.

Open formats reduce one form of lock-in. They do not guarantee identical behavior across engines. Implementations may support different format versions, data types, write operations, or maintenance features. Test the exact reader and writer combinations you will operate.

Do not adopt a lakehouse only to rename an existing file dump. The added value comes from reliable table state, governance, compatible engines, and disciplined operations.

A practical learning path

Learn object storage, analytical file formats, and data partitioning.
Trace one dataset from producer through raw, transformed, and curated states.
Study how a table format represents snapshots, manifests, and commits.
Practice schema and partition evolution with compatibility tests.
Compare query plans before and after partitioning, statistics, and compaction.
Define catalog metadata, ownership, and access rules for one table.
Automate snapshot retention, compaction, and orphan cleanup safely.
Test multiple engines before claiming cross-engine interoperability.
Design recovery, audit, cost, and freshness controls for the full platform.