Databricks Fundamentals

Databricks is a managed platform for data engineering, analytics, and artificial intelligence. It gives several roles one place to work with governed data. Data engineers build pipelines. Analysts run SQL and create dashboards. Data scientists explore data and train models. Administrators manage identities, compute, and access.

The central mental model is governed data plus separate compute plus shared workspace tools.

                         Databricks account
                                |
          +---------------------+---------------------+
          |                                           |
      workspace                                  Unity Catalog
 notebooks, jobs, SQL                         catalogs and permissions
          |                                           |
          +--------------------+----------------------+
                               |
                    compute reads and writes
                               |
                 Delta tables in cloud storage

This separation matters. A notebook is not a cluster. A table is not stored inside a notebook. Unity Catalog does not execute a query. Compute performs the work, storage keeps durable data, and governance controls access.

Why the platform exists

Data teams often assemble separate systems for ingestion, transformation, business intelligence, machine learning, governance, and scheduling. Those systems can create duplicate data, inconsistent access rules, and several operational boundaries.

Databricks uses the lakehouse pattern to bring data lake and data warehouse workloads closer together. Data remains in scalable cloud storage. Compute can scale separately. Delta Lake adds table reliability. Unity Catalog supplies a common governance layer. Workspace tools expose the platform to different roles.

The result is not one giant process. It is a set of connected layers with distinct jobs.

Start with account, workspace, and metastore

A Databricks account is the top-level administrative boundary. It can contain multiple workspaces. Account-level administration covers identities, workspace management, billing, and Unity Catalog metastores.

A workspace is a team environment. It contains working assets such as notebooks, SQL queries, dashboards, jobs, and files. A company may use one workspace or several workspaces for isolation among teams, environments, or regions.

A Unity Catalog metastore is the central governance system for data and artificial intelligence assets. A metastore can attach to multiple workspaces in the same region. Those workspaces can then share a governed view of registered assets, subject to permissions and workspace bindings.

Do not confuse the workspace browser with durable business data. Workspace storage holds working assets. Unity Catalog tables and volumes refer to data stored separately.

The control plane coordinates; the compute plane processes

Databricks separates platform services from workload execution.

The control plane contains Databricks-managed backend services and the web application. It coordinates workspace features.

The compute plane is where your code and queries process data. Its location depends on the compute type:

• serverless compute runs in a Databricks-managed serverless compute plane;
• classic compute runs in your cloud account and requires more infrastructure configuration;
• SQL warehouses provide compute optimized for SQL analytics and business intelligence.

This distinction affects networking, administration, startup behavior, supported features, and cost. Choose compute for the workload and security requirements. Do not treat every compute option as interchangeable.

Storage and compute have separate lifecycles

Durable data usually remains in cloud object storage while compute starts, scales, and stops independently. Terminating compute should not delete a governed table.

This separation lets different workloads use the same governed data. A scheduled pipeline can update a table. A SQL warehouse can query it later. A notebook can inspect it with different compute.

It also creates responsibilities. You must govern storage credentials and locations. You must select compute with the required language and feature support. You must stop or scale idle resources according to your cost policy.

Delta Lake makes files behave like reliable tables

Delta Lake is the default table format on Databricks. It extends Parquet data files with a file-based transaction log.

The transaction log supports atomic, consistent, isolated, and durable transactions. It also supports scalable metadata handling. Each committed write creates a new table version, which enables table history and queries against earlier versions while the required data files remain available.

Delta Lake also enforces table schemas on write. Operations such as update, delete, and merge can change table contents through transactions instead of direct file manipulation.

Use table operations rather than editing files inside a Delta table directory. Direct file changes bypass the transaction log and can damage the table's state.

Delta Lake does not remove every data-quality problem. A valid schema can still contain inaccurate values. Constraints, expectations, tests, monitoring, and ownership remain necessary.

Unity Catalog governs data and artificial intelligence assets

Unity Catalog is the platform's governance layer. It controls access, records lineage, supports discovery, and provides audit information for governed assets.

Its common data namespace has three levels:

catalog.schema.object

production.sales.orders

A catalog is the highest organizational level in that namespace. It contains schemas. A schema contains objects such as tables, views, volumes, functions, and models.

A securable object is an object on which privileges can be granted. Users, service principals, and groups are principals that can receive privileges. Prefer group-based grants so access follows roles instead of individual accounts.

Unity Catalog distinguishes managed and external assets. For a managed table or volume, Unity Catalog manages both governance and the underlying storage lifecycle. For an external table or volume, Unity Catalog governs access while the external location retains its independent lifecycle.

Use fully qualified names when context could be ambiguous. production.sales.orders makes the catalog and schema explicit. Relying on a session's current catalog or schema can make code behave differently in another environment.

Notebooks are development surfaces, not production boundaries

A Databricks notebook combines executable cells, results, visualizations, and explanatory text. You can work interactively with Python, SQL, Scala, or R when the attached compute supports that language.

Notebooks are useful for exploration, learning, debugging, and collaborative development. They can also serve as tasks inside a job.

Interactive success is not a production guarantee. A notebook may depend on cell order, hidden state, the current user, an attached compute configuration, or manually installed libraries. Production work needs explicit dependencies, parameters, source control, repeatable compute, and monitored runs.

Treat a notebook as source code. Keep each cell focused. Restart and run the notebook from a clean state. Move reusable logic into modules when it needs testing or reuse outside the notebook.

Compute runs notebooks, jobs, pipelines, and SQL

Databricks offers several compute families.

Serverless compute is allocated and managed by Databricks. It supports on-demand execution for supported notebooks, jobs, and pipelines without provisioning compute resources in your cloud account.

Classic compute is created and configured for workloads in your cloud account. It offers more direct infrastructure control and may be required for workloads that serverless compute does not support.

A SQL warehouse is compute for querying and exploring data with SQL. It powers interfaces such as the SQL editor, dashboards, and supported business intelligence connections.

Compute selection is a decision, not a badge of maturity:

Need	Starting point	Check before choosing
Interactive Python or SQL	Serverless notebook compute	Region, language, library, and feature support
Scheduled task	Serverless job compute	Task compatibility and network access
Custom runtime or network control	Classic compute	Policy, access mode, startup time, and administration
SQL analytics and dashboards	SQL warehouse	Size, concurrency, auto-stop, and permissions

Permissions exist at more than one layer. A user may have permission to use compute but lack SELECT on a table. Another user may have table access but lack permission to attach to compute. Diagnose both resource access and data access.

Databricks Runtime defines the execution environment

Databricks Runtime packages Apache Spark and other components into a tested execution environment for classic compute. Runtime versions differ in Spark version, bundled libraries, features, and support lifecycle. Long-term support releases have extended support windows.

Serverless compute uses Databricks-managed runtime versions rather than a classic cluster runtime selected by the user.

Pin and test runtime assumptions for production workloads. An interactive notebook that works on a newer runtime may fail on an older job cluster. A library that modifies Spark internals may behave differently after an upgrade.

Apache Spark and SQL are execution interfaces

Databricks is built on Apache Spark for distributed data processing. You can use Spark DataFrames from Python, Scala, Java, or R, and you can use SQL for relational work.

SQL and DataFrame operations can read the same governed tables. The best interface depends on the task and team. SQL is often clear for relational transformations and analytics. DataFrame APIs fit programmatic pipelines and application logic.

Do not mistake the interface for the platform. Spark performs distributed computation. Delta Lake supplies table storage semantics. Unity Catalog governs assets. Databricks connects these layers and adds managed services around them.

Lakeflow Jobs turns work into repeatable runs

Lakeflow Jobs schedules and orchestrates work on Databricks. A job contains one or more tasks. Tasks can run notebooks, Python files, pipelines, SQL, and other supported workloads. Dependencies form a directed acyclic graph.

A trigger starts a job. Triggers can be time-based or event-based. A manual or external call can also start a run.

Production jobs need more than a schedule:

• a service principal or defined run identity;
• parameters instead of edited constants;
• task dependencies that match data dependencies;
• retries and timeouts appropriate to side effects;
• notifications and run monitoring;
• controlled source code and deployment;
• idempotent outputs when a task can run again.

The job run view exposes task status, duration, logs, metrics, and dependencies. Use that evidence to locate the failing task before changing unrelated compute settings.

The medallion pattern organizes refinement

The medallion architecture is a design pattern for incrementally refining data through layers. Teams often call these bronze, silver, and gold:

sources -> bronze -> silver -> gold -> dashboards, applications, and models
           raw       cleaned    business-ready

Bronze preserves source fidelity. Silver applies validation, deduplication, and standardization. Gold presents data for a defined business or analytical purpose.

These names are conventions, not enforced product objects. A catalog, schema, or table does not become correct because its name contains silver. Define the quality contract, ownership, and consumers for each layer.

Do not copy every dataset through three layers by reflex. Use the pattern when staged refinement improves traceability, reuse, or quality management.

A representative flow

Consider an order analytics workload:

1. A pipeline ingests source files into a bronze Delta table.
2. A transformation validates identifiers and deduplicates records into a silver table.
3. Another transformation aggregates daily revenue into a gold table.
4. Unity Catalog controls which principals can read each table and records lineage.
5. Lakeflow Jobs schedules the tasks and records each run.
6. A SQL warehouse serves the gold table to a dashboard.
7. Engineers inspect failed tasks and query history when results are late or incorrect.

The same data crosses several platform layers. Storage holds the tables. Compute performs each transformation and query. Unity Catalog governs access. The workspace exposes code, job configuration, and monitoring.

Where Databricks fits well

Databricks fits work that benefits from shared, governed data across several analytical workloads. Common uses include batch and streaming data engineering, SQL analytics, business intelligence, data science, machine learning, and governed data sharing.

It can reduce boundaries among these workloads when they need the same tables, identity model, and operational tooling.

Limits and poor fits

Databricks is not automatically the best system for every data task.

A small local transformation may not justify a distributed platform. An operational application that needs high-rate, low-latency row transactions may need an online transaction processing database. A team with a mature warehouse and no cross-workload need may gain little from migration.

Managed services also retain operational choices. You still design schemas, permissions, pipelines, tests, cost controls, and recovery procedures. Serverless compute removes infrastructure management for supported workloads, but it does not remove workload design or cost.

Platform features and names change over time. Confirm current cloud, region, runtime, and compute support in official documentation before making an architecture commitment.

A path from fundamentals to competence

1. Learn the account, workspace, metastore, storage, and compute boundaries.
2. Open Catalog Explorer and trace one fully qualified table name.
3. Run a SQL query against a sample table and identify the compute that executed it.
4. Create a notebook that reads a governed table and produces a small result.
5. Inspect the table's schema, history, permissions, and lineage.
6. Compare managed and external Unity Catalog objects.
7. Turn a clean notebook into a parameterized Lakeflow Jobs task.
8. Monitor a run and locate its task logs and compute details.
9. Build a small bronze-to-silver-to-gold flow with explicit quality rules.
10. Study cloud architecture, security, networking, cost controls, deployment automation, and disaster recovery before production ownership.