openskills.info
Course catalog
Apache Flink Fundamentals logoCourse Preview

Apache Flink Fundamentals

Apache Flink is a distributed stream-processing framework that handles both real-time event streams and batch workloads. It provides exactly-once state consistency, event-time processing, and windowing for continuous data computations at scale.

itData engineering and analytics

Apache Flink Fundamentals

Apache Flink is a distributed engine for stateful computations over bounded and unbounded data streams. You describe a dataflow. Flink runs its operators in parallel, moves records between them, manages application state, and recovers that state after failures.

The central mental model is a continuously running dataflow with durable memory.

sources -> transformations -> keyed state and time -> sinks
             |                      |
             +---- parallel tasks --+
                         |
                  periodic checkpoints

That model separates Flink from a message broker. A broker stores and transports records. Flink computes over records. It also separates Flink from a database. A database serves stored data through queries and transactions. Flink keeps computation-specific state while data moves through a dataflow.

Why stream processing exists

Many systems produce records continuously: transactions, telemetry, logs, clicks, inventory changes, and database change events. A periodic batch job waits for a boundary, reads a finite input, computes a result, and stops. That design is effective when delay is acceptable and the input has a useful boundary.

A streaming job consumes records as they arrive. It can update a result, enrich an event, detect a pattern, or trigger an action without waiting for the next batch. The harder part is not reading one record. The harder part is remembering prior records, reasoning about time, handling disorder, and recovering without corrupting the result.

Flink puts those concerns inside one processing engine. It supports three broad application families:

  • continuous data pipelines that transform and move records;
  • streaming and batch analytics that calculate and update results;
  • event-driven applications that change state or trigger actions in response to events.

Bounded and unbounded streams

An unbounded stream has a beginning but no defined end. A job cannot wait for every record because the complete input never arrives. It must produce useful results incrementally.

A bounded stream has a defined end. Flink can process it as a finite input and finish. This is batch processing expressed through the same dataflow model.

Flink's DataStream API has streaming and batch runtime modes. Streaming mode supports continuous incremental processing and is required for unbounded jobs. Batch mode can use optimizations that depend on a known finite input. With bounded input, both modes produce the same final result when that result is interpreted correctly. Their intermediate output and execution behavior can differ.

The unified model is useful when the same logic must process live records and historical data. It does not make bounded and unbounded operation identical. Infinite input changes how you define completeness, joins, aggregation, state retention, and output updates.

A Flink application is a graph

Continue the course

This section is part of the paid course.

See pricing to subscribe, or log in if you already have access.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Sources