Apache Spark Fundamentals

Apache Spark executes data processing work across one machine or a cluster. You describe transformations over distributed data. Spark plans the work, divides it into tasks, and runs those tasks in parallel.

The central mental model is a lazy dataflow executed over partitions.

data sources -> transformations -> logical plan -> jobs -> stages -> tasks
                                                        |          |
                                                     shuffles   partitions
                                                                   |
                                                               executors

Spark is a compute engine. It reads from storage systems and writes results back to them. It does not replace object storage, a distributed file system, a warehouse, or a transactional database.

Why distributed data processing exists

One process eventually meets a limit. The input may exceed one machine's memory. A computation may take too long on one core. Data may already live across many files or machines.

Distributed processing divides the input into partitions. Several workers can process different partitions at the same time. This adds capacity, but it also adds network transfer, serialization, scheduling, and failure modes.

Spark gives you one programming model for this work. You can express relational transformations with DataFrames or SQL. You can use lower-level resilient distributed datasets when you need direct control over distributed objects. Structured Streaming extends the structured APIs to incremental queries over arriving data.

One application has one driver and many executors

A Spark application consists of a driver process and executor processes.

The driver runs your main program. It creates the Spark session, builds execution plans, requests resources, schedules work, and collects status. The driver must remain reachable from the executors during the application.

A cluster manager allocates resources. Spark supports its standalone manager, Hadoop YARN, and Kubernetes.

An executor belongs to one application. It runs tasks and can keep data in memory or on disk for that application.

                         cluster manager
                               |
driver -> request resources ---+
   |
   +------ tasks ------> executor A -> partitions 0 and 1
   +------ tasks ------> executor B -> partitions 2 and 3

Applications do not share executor memory. To share data across applications, write it to an external storage system.

Start with SparkSession

SparkSession is the entry point for Spark functionality in current structured applications. It creates DataFrames, runs SQL, reads data, writes data, and exposes the underlying Spark context.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("orders-summary").getOrCreate()

orders = spark.read.parquet("s3a://warehouse/orders/")
result = orders.filter("status = 'paid'").groupBy("country").count()
result.write.mode("overwrite").parquet("s3a://warehouse/orders-by-country/")

spark.stop()

The builder does not need to hardcode a cluster address. Supply deployment settings through spark-submit so the same application can run locally or through a cluster manager.

DataFrames are the default structured abstraction

A DataFrame is a distributed dataset organized into named columns. Its schema tells Spark about the structure of the data and the operations you want to perform. Spark SQL uses that information to optimize execution.

SQL and DataFrame operations use the same Spark SQL engine. Choose the interface that expresses the transformation clearly. You can register a DataFrame as a temporary view, query it with SQL, and continue with DataFrame operations.

In Scala and Java, a Dataset adds a typed object interface while retaining the structured engine. Python and R expose DataFrames rather than the typed Dataset API.

An RDD, or resilient distributed dataset, is the lower-level distributed collection. It contains elements split across partitions and supports parallel operations. RDDs remain useful when the data or algorithm does not fit the structured APIs. For tabular and relational work, DataFrames usually give Spark more information to optimize.

Transformations are lazy

A transformation describes a new dataset from an existing dataset. Examples include filter, select, map, join, and group operations.

An action requests a result or writes output. Examples include count, collect, show, and save operations.

Spark does not immediately execute each transformation. It records the operations and waits for an action. The action creates a job.

read -> filter -> select -> group -> write
         lazy      lazy      lazy    action
                                      |
                                      +-> job starts

Lazy evaluation gives Spark a view of the dataflow before execution. It also explains why a program can build several DataFrames without doing cluster work.

Be careful with actions that return data to the driver. collect() can overwhelm driver memory when the result is large. Prefer distributed writes or bounded inspection methods when you do not know the result size.

Partitions define task-level parallelism

A partition is one logical slice of a distributed dataset. Spark normally creates one task for each partition in a stage. More partitions can expose more parallel work. Too many small partitions create scheduling overhead. Too few large partitions leave cores idle and increase the cost of a slow task.

Partition count is not the same as executor count. Executors provide process and resource capacity. Partitions provide units of data parallelism. Tasks connect the two at runtime.

Data distribution matters as much as partition count. A heavily skewed key can place much more data in one partition than in its peers. The stage then finishes at the speed of the slow partition.

Shuffles create stage boundaries

A shuffle redistributes data across partitions. Grouping by key, joining unrelated partitions, sorting, and repartitioning can require a shuffle.

before shuffle                after shuffle

executor A: red, blue  ----\  executor A: all blue
executor B: blue, red  -----X executor B: all red
executor C: red, blue  ----/  executor C: another key range

Shuffle data can cross executors and machines. Spark may also spill intermediate structures to disk when they do not fit in memory. Network transfer, disk input and output, serialization, and skew make shuffles a common performance boundary.

A job is divided into stages around shuffle dependencies. Each stage contains tasks that can run without another shuffle boundary.

Do not treat every shuffle as a defect. Aggregation and joins often require data movement. Inspect the execution plan and the Spark user interface, then reduce unnecessary movement or fix skew with evidence.

Cache only data you will reuse

Without persistence, Spark may recompute a transformed dataset for each action. cache() or persist() marks computed partitions for reuse.

Caching helps when multiple actions reuse an expensive intermediate dataset. It also consumes executor memory or disk and adds materialization cost. A cached DataFrame does not appear as materialized storage until an action computes it.

Unpersist data when reuse ends. Do not cache every intermediate result. Recomputing a cheap pipeline can cost less than storing and evicting it.

Resilience comes from recoverable computation

RDDs record how they were derived. If a cached partition is lost, Spark can recompute it from the transformations that created it. Failed tasks can be retried according to Spark's failure policy.

This recovery model does not make arbitrary side effects safe. A task may run more than once. Writes to external systems need commit protocols, transactions, idempotency, or another duplicate-control mechanism appropriate to that system.

The driver is also a special failure boundary. Executors depend on it for scheduling and coordination. Deployment mode and cluster-manager features determine how the driver starts and whether it can be restarted.

Batch and streaming share structured APIs

Structured Streaming models an input stream as an input table. A query incrementally updates a result table as new rows arrive. You write the query with DataFrame or Dataset operations and start it as a streaming query.

The default engine uses micro-batches. Spark records progress and state in a checkpoint location so a compatible query can recover after a failure.

Output mode controls what reaches the sink:

append writes new result rows that will not change;
update writes result rows changed since the last trigger;
complete writes the full result table after a trigger.

The query shape and sink determine which modes are valid. End-to-end exactly-once behavior also depends on replayable sources and a sink with compatible fault-tolerance semantics. Some sinks provide only at-least-once behavior. Arbitrary foreachBatch logic provides only the guarantees you implement.

Watermarks let stateful event-time queries bound how long they wait for late data and when they can remove old state. A watermark is a policy threshold. Data later than that threshold may be dropped, depending on the operation.

Continuous processing is an experimental alternative for a limited set of streaming queries. It trades the default micro-batch guarantee for lower latency and at-least-once fault tolerance.

Submit the application, not a machine-specific script

spark-submit launches packaged applications through a uniform interface.

./bin/spark-submit \
  --master k8s://https://api.example:6443 \
  --deploy-mode cluster \
  --conf spark.executor.instances=8 \
  --conf spark.executor.memory=4g \
  app.py

The master selects local execution or a cluster manager. The deploy mode selects where the driver runs. In client mode, the submitting process hosts the driver. In cluster mode, the cluster launches the driver.

Use local mode for learning and tests that do not need real distributed failure behavior. A successful local run does not validate network reachability, distributed storage paths, serialization, resource contention, or cluster security.

Observe the plan and the runtime

The Spark user interface exposes jobs, stages, tasks, executors, storage, SQL plans, and streaming progress. The Jobs page shows the job's directed acyclic graph and stages. The Stages page exposes task duration, shuffle metrics, and skew. The SQL page connects structured queries to their physical plans.

The live application user interface normally belongs to the driver. Event logging and the history server preserve the information after an application ends.

Start performance work with evidence:

confirm the physical plan;
find the slow stage;
compare task durations and input sizes;
inspect shuffle reads and writes;
check spill, garbage collection, and executor loss;
check whether the driver collects too much data;
verify that the source and sink can sustain the requested throughput.

Spark programs can be limited by CPU, memory, network bandwidth, disk, serialization, data skew, or an external system. Executor memory is not a universal fix.

Where Spark fits well

Spark is a good fit when one computation benefits from distributed parallelism and can be expressed as transformations over partitioned data. Common uses include:

large batch transformations and aggregations;
SQL analysis over files, tables, and supported data sources;
iterative analytics that reuse intermediate datasets;
machine-learning pipelines through Spark libraries;
stateful incremental processing with Structured Streaming;
one application that combines structured batch and streaming logic.

Limits and poor fits

Spark adds a distributed runtime. It can be excessive for small inputs that one process handles comfortably.

It is not a transactional system of record. It does not replace the storage and serving systems around it. Row-by-row request handling, low-latency point lookups, and fine-grained transactions usually belong elsewhere.

Distributed execution also does not repair a poor algorithm or bad data layout. A full shuffle over skewed input can remain slow on a larger cluster. More executors can increase contention against a limited source or sink.

Choose Spark when the parallel dataflow justifies its operational cost. Choose a simpler tool when one machine or the target database can perform the work directly.

A path from fundamentals to competence

Run the official quick start in local mode and inspect a DataFrame schema.
Build a transformation chain and predict which operation starts a job.
Open the user interface and connect one action to its jobs, stages, and tasks.
Compare a narrow transformation with a shuffled aggregation.
Reuse one expensive DataFrame with and without caching.
Package the application and launch it with spark-submit.
Read a physical plan and relate it to shuffle, partition, and join behavior.
Build a checkpointed Structured Streaming query with a fault-tolerant sink.
Test failure recovery and external side-effect behavior.
Study tuning, security, monitoring, and cluster-specific deployment guidance before production work.