Databricks Fundamentals
Databricks is a unified analytics platform built on Apache Spark that combines data engineering, data science, and machine learning in a collaborative workspace. It provides managed Spark clusters, a lakehouse architecture, notebooks, and workflow orchestration.
itData engineering and analytics | OpenSkills.info
Intro
Databricks Fundamentals
Databricks is a managed platform for data engineering, analytics, and artificial intelligence. It gives several roles one place to work with governed data. Data engineers build pipelines. Analysts run SQL and create dashboards. Data scientists explore data and train models. Administrators manage identities, compute, and access.
The central mental model is governed data plus separate compute plus shared workspace tools.
Databricks account
|
+---------------------+---------------------+
| |
workspace Unity Catalog
notebooks, jobs, SQL catalogs and permissions
| |
+--------------------+----------------------+
|
compute reads and writes
|
Delta tables in cloud storage
This separation matters. A notebook is not a cluster. A table is not stored inside a notebook. Unity Catalog does not execute a query. Compute performs the work, storage keeps durable data, and governance controls access.
Why the platform exists
Data teams often assemble separate systems for ingestion, transformation, business intelligence, machine learning, governance, and scheduling. Those systems can create duplicate data, inconsistent access rules, and several operational boundaries.
Databricks uses the lakehouse pattern to bring data lake and data warehouse workloads closer together. Data remains in scalable cloud storage. Compute can scale separately. Delta Lake adds table reliability. Unity Catalog supplies a common governance layer. Workspace tools expose the platform to different roles.
The result is not one giant process. It is a set of connected layers with distinct jobs.
Start with account, workspace, and metastore
A Databricks account is the top-level administrative boundary. It can contain multiple workspaces. Account-level administration covers identities, workspace management, billing, and Unity Catalog metastores.
A workspace is a team environment. It contains working assets such as notebooks, SQL queries, dashboards, jobs, and files. A company may use one workspace or several workspaces for isolation among teams, environments, or regions.
A Unity Catalog metastore is the central governance system for data and artificial intelligence assets. A metastore can attach to multiple workspaces in the same region. Those workspaces can then share a governed view of registered assets, subject to permissions and workspace bindings.
Do not confuse the workspace browser with durable business data. Workspace storage holds working assets. Unity Catalog tables and volumes refer to data stored separately.
The control plane coordinates; the compute plane processes
Databricks separates platform services from workload execution.
The control plane contains Databricks-managed backend services and the web application. It coordinates workspace features.
The compute plane is where your code and queries process data. Its location depends on the compute type:
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://docs.databricks.com/aws/en/introduction/
Supports
- Databricks as a managed platform for enterprise data, analytics, and artificial intelligence workloads
- Common use cases across lakehouse engineering, SQL analytics, business intelligence, machine learning, and job orchestration
- Notebook language support and workspace tools
- Managed open source integrations including Apache Spark, Delta Lake, MLflow, and Unity Catalog
- https://docs.databricks.com/aws/en/getting-started/concepts
Supports
- Account and workspace responsibilities
- Users, service principals, groups, and access-control concepts
- Databricks unit billing concept
- https://docs.databricks.com/aws/en/getting-started/high-level-architecture
Supports
- Account, workspace, and Unity Catalog metastore relationships
- Three-level Unity Catalog namespace
- Control-plane and compute-plane responsibilities
- Serverless and classic compute-plane locations on AWS
- Separation of workspace files from governed data objects
- Workspaces as environments for ingestion, exploration, jobs, and machine-learning workloads
- https://docs.databricks.com/aws/en/getting-started
Supports
- Official practice path for querying and visualizing data
- Creating governed tables through Unity Catalog
- Data-engineering and machine-learning tutorials
- https://docs.databricks.com/aws/en/lakehouse/
Supports
- Lakehouse as a combination of data lake and data warehouse capabilities
- Compute decoupled from storage through Apache Spark
- Delta Lake and Unity Catalog roles in the Databricks lakehouse
- Shared data across data engineering, analytics, business intelligence, and machine-learning workloads
- Medallion architecture as incremental refinement through data layers
- https://docs.databricks.com/aws/en/delta
Supports
- Delta Lake as the default Databricks table format
- Parquet data files plus a file-based transaction log
- Atomic transactions, scalable metadata, and schema enforcement
- Table update, delete, merge, history, and previous-version queries
- Spark and Structured Streaming integration
- Warning against direct changes to transaction-log and data files
- Cloud object storage for Delta table data and metadata
- https://docs.databricks.com/aws/en/data-governance/unity-catalog/
Supports
- Unity Catalog as the governance layer for data and artificial intelligence
- Three-level catalog, schema, and object namespace
- Securable objects and privileges for users, service principals, and groups
- Access control, discovery, lineage, auditing, classification, monitoring, and sharing capabilities
- Managed and external asset distinction
- https://docs.databricks.com/aws/en/catalogs
Supports
- Catalog as the first level and primary organizational unit in Unity Catalog
- Catalogs containing schemas and schemas containing governed objects
- Catalog design as an isolation and access boundary
- https://docs.databricks.com/aws/data-governance/unity-catalog/paths
Supports
- Fully qualified table identifiers
- Managed and external table and volume storage behavior
- Unity Catalog management of managed storage paths
- Governance of external object paths and non-overlap requirements
- https://docs.databricks.com/aws/en/compute
Supports
- Serverless, classic, and SQL warehouse compute families
- Serverless compute as on-demand managed resources
- Classic compute as configurable provisioned resources
- SQL warehouses as optimized compute for SQL analytics
- https://docs.databricks.com/aws/en/compute/serverless
Supports
- Databricks management of serverless resource allocation and scaling
- Supported serverless notebook, job, and pipeline workloads
- Need to check serverless requirements and limitations
- Separate configuration paths for other serverless platform features
- https://docs.databricks.com/aws/en/compute/use-compute
Supports
- Classic compute resources deployed in the customer's cloud account
- Classic compute permissions such as attach, restart, and manage
- Difference between resource permission and data-object permission
- https://docs.databricks.com/aws/en/compute/sql-warehouse
Supports
- SQL warehouse as compute for querying and exploring Databricks data
- SQL editor, Catalog Explorer, dashboard, and business intelligence use
- Serverless warehouse recommendation where available
- Start, auto-start, sizing, scaling, auto-stop, and permission concerns
- https://docs.databricks.com/aws/en/release-notes/runtime
Supports
- Runtime releases packaging Apache Spark versions and Databricks components
- Release support dates and long-term support variants
- Separate Databricks-managed release process for serverless compute
- https://docs.databricks.com/aws/en/notebooks/run-notebook
Supports
- Interactive execution of one or more notebook cells
- Attachment to compute before cell execution
- Run-all and selected-cell behavior
- Scheduling notebooks as jobs
- https://docs.databricks.com/aws/en/notebooks/notebooks-code
Supports
- Notebook development and code-editor behavior
- Mixed notebook development interfaces and SQL result integration
- Source-oriented notebook development practices
- https://docs.databricks.com/aws/en/jobs/
Supports
- Lakeflow Jobs as orchestration for data-processing workloads
- Jobs, tasks, dependencies, triggers, parameters, notifications, and run monitoring
- Notebook, pipeline, Python, SQL, and other supported task types
- Scheduled, event-based, manual, and external execution
- Programmatic management through command-line tools, bundles, software-development kits, and application interfaces
- https://docs.databricks.com/aws/en/jobs/jobs-quickstart
Supports
- Two-task notebook workflow using governed data
- Serverless job compute default when supported
- Task parameters, immediate runs, and run-result inspection
- https://docs.databricks.com/aws/en/sql/get-started/data-warehousing-concepts
Supports
- Databricks SQL and SQL warehouse roles in lakehouse analytics
- SQL compute separated from storage
- Unity Catalog governance and Delta Lake table reliability for warehouse workloads
- Medallion data refinement for analytical modeling