Apache Spark Fundamentals

Apache Spark executes data processing work across one machine or a cluster. You describe transformations over distributed data. Spark plans the work, divides it into tasks, and runs those tasks in parallel.

The central mental model is a lazy dataflow executed over partitions.

data sources -> transformations -> logical plan -> jobs -> stages -> tasks
                                                        |          |
                                                     shuffles   partitions
                                                                   |
                                                               executors

Spark is a compute engine. It reads from storage systems and writes results back to them. It does not replace object storage, a distributed file system, a warehouse, or a transactional database.

Why distributed data processing exists

One process eventually meets a limit. The input may exceed one machine's memory. A computation may take too long on one core. Data may already live across many files or machines.

Distributed processing divides the input into partitions. Several workers can process different partitions at the same time. This adds capacity, but it also adds network transfer, serialization, scheduling, and failure modes.

Spark gives you one programming model for this work. You can express relational transformations with DataFrames or SQL. You can use lower-level resilient distributed datasets when you need direct control over distributed objects. Structured Streaming extends the structured APIs to incremental queries over arriving data.

One application has one driver and many executors

A Spark application consists of a driver process and executor processes.

The driver runs your main program. It creates the Spark session, builds execution plans, requests resources, schedules work, and collects status. The driver must remain reachable from the executors during the application.

A cluster manager allocates resources. Spark supports its standalone manager, Hadoop YARN, and Kubernetes.

An executor belongs to one application. It runs tasks and can keep data in memory or on disk for that application.

                         cluster manager
                               |
driver -> request resources ---+
   |
   +------ tasks ------> executor A -> partitions 0 and 1
   +------ tasks ------> executor B -> partitions 2 and 3

Applications do not share executor memory. To share data across applications, write it to an external storage system.

Apache Spark Fundamentals

Apache Spark Fundamentals

Why distributed data processing exists

One application has one driver and many executors

Start with SparkSession

Relevant careers