Apache Flink Fundamentals

Apache Flink is a distributed engine for stateful computations over bounded and unbounded data streams. You describe a dataflow. Flink runs its operators in parallel, moves records between them, manages application state, and recovers that state after failures.

The central mental model is a continuously running dataflow with durable memory.

sources -> transformations -> keyed state and time -> sinks
             |                      |
             +---- parallel tasks --+
                         |
                  periodic checkpoints

That model separates Flink from a message broker. A broker stores and transports records. Flink computes over records. It also separates Flink from a database. A database serves stored data through queries and transactions. Flink keeps computation-specific state while data moves through a dataflow.

Why stream processing exists

Many systems produce records continuously: transactions, telemetry, logs, clicks, inventory changes, and database change events. A periodic batch job waits for a boundary, reads a finite input, computes a result, and stops. That design is effective when delay is acceptable and the input has a useful boundary.

A streaming job consumes records as they arrive. It can update a result, enrich an event, detect a pattern, or trigger an action without waiting for the next batch. The harder part is not reading one record. The harder part is remembering prior records, reasoning about time, handling disorder, and recovering without corrupting the result.

Flink puts those concerns inside one processing engine. It supports three broad application families:

• continuous data pipelines that transform and move records;
• streaming and batch analytics that calculate and update results;
• event-driven applications that change state or trigger actions in response to events.

Bounded and unbounded streams

An unbounded stream has a beginning but no defined end. A job cannot wait for every record because the complete input never arrives. It must produce useful results incrementally.

A bounded stream has a defined end. Flink can process it as a finite input and finish. This is batch processing expressed through the same dataflow model.

Flink's DataStream API has streaming and batch runtime modes. Streaming mode supports continuous incremental processing and is required for unbounded jobs. Batch mode can use optimizations that depend on a known finite input. With bounded input, both modes produce the same final result when that result is interpreted correctly. Their intermediate output and execution behavior can differ.

The unified model is useful when the same logic must process live records and historical data. It does not make bounded and unbounded operation identical. Infinite input changes how you define completeness, joins, aggregation, state retention, and output updates.

A Flink application is a graph

A Flink program defines a directed dataflow graph:

1. A source reads records from an external system or bounded collection.
2. A transformation applies logic such as mapping, filtering, grouping, joining, or aggregating.
3. A sink sends results to an external system.

Flink converts the program into an execution graph. Each operator can run as several parallel subtasks. Records move between subtasks according to the dataflow's partitioning rules.

Parallelism controls how many subtasks execute an operator. More parallelism can increase capacity when enough resources and input partitions exist. It also increases coordination, network exchange, and state partitioning. A higher number is not an automatic performance fix.

Flink can chain compatible operators into one task. Chaining reduces thread handoffs and serialization between adjacent operators. A network shuffle remains necessary when records must be repartitioned, such as before keyed processing.

Keys put related records together

Many computations require all records for one logical entity to meet the same state. Examples include an account balance, a device's latest status, or a per-customer fraud rule.

A keyBy operation partitions records by a key. Flink routes the same key to the same parallel instance of the next keyed operator. That operator can then access keyed state scoped to the current key.

record(account=42) -- keyBy(account) --> subtask responsible for account 42
                                              |
                                      state for account 42

Flink also supports operator state, which is scoped to a parallel operator instance rather than a record key. Source offsets and partition assignments are common examples of operator-level information.

Managed state is part of the runtime contract. Flink controls its storage, snapshots it, and redistributes keyed state when a compatible job is rescaled. State still has a cost. Unbounded keys, windows, joins, or deduplication rules can create unbounded state unless you define cleanup or retention.

Time is data, not just a clock

Distributed records rarely arrive in the order in which their real-world events occurred. Network delay, retries, partitions, and offline devices all introduce disorder.

Flink distinguishes two core notions of time:

• Processing time uses the clock of the machine executing an operator. It is low overhead, but results can change with delays and restarts.
• Event time uses a timestamp carried by each record. It lets a computation reason about when the event occurred.

A watermark is Flink's progress signal for event time. A watermark at a time tells downstream operators that event time has advanced to that point. Watermarks let a window decide when it has enough evidence to produce a result.

Watermarks are not proof that no older event will ever arrive. A late event can arrive behind the watermark. You must choose a policy: discard it, accept it for a limited period and update a result, or route it elsewhere.

This creates a real tradeoff. Waiting longer can include more delayed events, but it delays results and retains state longer. Advancing quickly lowers latency, but more records become late.

In a parallel dataflow, the slowest active input watermark limits downstream event-time progress. An idle partition can therefore stall windows unless the source or watermark strategy marks it idle.

Windows make infinite input finite enough to aggregate

You cannot compute a final count of an unbounded stream. You can compute a count for a defined scope.

A window assigns records to a finite group. Common forms include:

• tumbling windows with adjacent, non-overlapping intervals;
• sliding windows with overlapping intervals;
• session windows separated by a period of inactivity;
• count windows based on a number of records rather than elapsed time.

The window definition answers which records belong together. A trigger decides when the operator evaluates the window. Allowed lateness and late-data handling decide what happens after the event-time watermark passes the window boundary.

Windows are not free storage. A windowed computation holds state until the runtime can clear it. Large windows, many keys, long lateness, or skewed keys can increase state and checkpoint cost.

Checkpoints make state recoverable

Flink periodically creates a consistent snapshot of managed state and corresponding source positions. That snapshot is a checkpoint.

After a failure, Flink restores operator state from a completed checkpoint. Replayable sources resume from the recorded positions. The dataflow reprocesses records that arrived after that snapshot.

This mechanism can provide exactly-once consistency for Flink-managed state. The phrase does not mean that each physical record is executed only once. Recovery can replay records. It means replay does not make the restored managed state reflect the same logical input twice.

End-to-end exactly-once delivery needs more than Flink state. The source must participate in snapshotting and support recovery from a recorded position. The sink must coordinate its external writes with checkpoints or provide an equivalent transactional or idempotent contract. A sink that performs an irreversible side effect on every attempt can still duplicate that side effect after replay.

Checkpointing is disabled by default for streaming applications. Production jobs must configure checkpoint intervals, durable storage, and restart behavior. Checkpoint frequency trades runtime overhead against recovery work. State size, backpressure, storage throughput, and alignment behavior all affect checkpoint duration.

Checkpoints and savepoints have different jobs

A checkpoint is runtime-managed recovery material. Flink creates checkpoints frequently and manages their lifecycle for failure recovery.

A savepoint is user-managed application state for planned operations. You trigger and retain it when you intend to stop, upgrade, migrate, or rescale a compatible application.

Do not treat either artifact as a generic database backup. Restore compatibility depends on stable operator identity, compatible state schemas, supported Flink versions, and the snapshot format. Test the actual upgrade and restore path before relying on it.

The runtime has coordinators and workers

A Flink cluster has a JobManager process and one or more TaskManager processes.

The JobManager coordinates distributed execution. Its responsibilities include scheduling tasks, coordinating checkpoints, handling failures, and managing recovery. A JobMaster manages one job's execution. Other JobManager components expose submission interfaces and manage cluster resources.

A TaskManager is a worker. It executes subtasks and exchanges records with other TaskManagers.

A task slot is the smallest unit of resource scheduling in a TaskManager. Slots separate the TaskManager's managed resources for concurrent work. Multiple operators from the same job can share a slot through slot sharing. A slot is not the same as a CPU core, thread, or container.

Flink supports two main cluster lifecycles:

• Application Mode runs a cluster for one application, providing application-level isolation.
• Session Mode lets multiple jobs share an existing cluster.

Flink can run in standalone environments and integrate with resource managers such as Kubernetes and YARN. Deployment mode does not remove the need to size TaskManagers, configure durable checkpoint storage, secure interfaces, and monitor the job.

Choose the API from the problem

Flink offers several abstraction levels.

The DataStream API exposes streams, transformations, partitioning, state, timers, and detailed control over event processing. Choose it for custom event logic and explicit control of the dataflow.

The Table API expresses relational operations in Java or Python. Flink SQL expresses relational logic in SQL. Both use Flink's table runtime and optimizer. They are often the shorter path for filters, projections, joins, aggregations, and continuous table pipelines.

Tables over unbounded input can change over time. A continuous query can therefore emit inserts, updates, and deletes rather than one immutable final table. The sink must understand the query's changelog behavior.

You can convert between tables and DataStreams for mixed applications. Use that boundary deliberately. Crossing APIs adds type, schema, and update-semantics decisions that the team must understand.

Backpressure is a system signal

If a downstream task consumes records more slowly than an upstream task produces them, buffers fill and pressure propagates upstream. This is backpressure.

Backpressure protects the job from unlimited in-memory buffering. It also indicates that throughput is constrained somewhere downstream. The cause may be a slow sink, skewed key, expensive operator, insufficient parallelism, external service latency, or saturated network and storage.

Adding resources without locating the bottleneck can move the problem. Inspect task metrics, busy and backpressured time, record rates, checkpoint behavior, state size, and external dependencies together.

Where Flink fits well

Flink is a strong fit when a computation needs several of these properties together:

• continuous processing with low delay;
• state that spans many records;
• event-time reasoning over delayed or out-of-order records;
• consistent recovery of distributed application state;
• high parallel throughput;
• the same core logic over live and bounded historical input.

Representative uses include continuous extract-transform-load pipelines, fraud and anomaly detection, rule-based alerting, real-time metrics, stream enrichment, incremental materialization, and change-data-capture processing.

Limits and poor fits

Flink is an engine, not a complete data platform. You still need durable sources and sinks, schemas, deployment automation, security, observability, capacity planning, and an upgrade strategy.

Flink also adds operational weight. A short stateless transformation may be easier in a broker-native processor, database, or scheduled job. An interactive query engine may be a better fit for ad hoc exploration. A database remains the right system of record for transactional access.

Exactly-once state consistency does not repair a nontransactional sink. Event time does not tell you how late is acceptable. Checkpoints do not replace testing. More parallelism does not fix a single hot key. The engine supplies mechanisms; you supply the domain policy and operating discipline.

A practical learning path

1. Run a local cluster and inspect one example job in the Web UI.
2. Build a source, transformation, and sink with the DataStream API.
3. Partition by key and add small managed state with a clear cleanup rule.
4. Assign event timestamps, generate watermarks, and test late records.
5. Compare streaming and batch execution on bounded input.
6. Enable checkpoints and practice failure recovery from durable storage.
7. Create and restore a savepoint with stable operator identifiers.
8. Express the same transformation with Table API or SQL and inspect its changelog.
9. Measure backpressure, state growth, checkpoint duration, and recovery time.
10. Deploy through an application-specific lifecycle and rehearse upgrades.