Stream Processing
Stream processing continuously transforms data as records arrive. It lets you calculate, join, detect, or route information before waiting for a complete batch.
itData engineering and analytics | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Intro
Stream Processing
Stream processing is computation over data that keeps arriving. A payment, sensor reading, click, or database change enters a pipeline as a record. The pipeline transforms it and may produce a result straight away.
The important shift is that the input may never end. A batch job can wait for a complete dataset. A stream processor must decide what a useful result means while more records are still on the way.
The processing model
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/overview/
Supports
- Stateful and timely stream processing concepts
- Transformations, joins, aggregations, windows, and state in streaming APIs
- https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/stateful-stream-processing/
Supports
- Stateful operations and keyed state
- Checkpoints, input replay, recovery, and fault-tolerance boundaries
- https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream-v2/builtin-funcs/windows/
Supports
- Windows as finite buckets for infinite streams
- Tumbling, sliding, session, event-time, and processing-time windows
- https://beam.apache.org/documentation/programming-guide/
Supports
- Event time, watermarks, windowing, triggers, and late data
- https://kafka.apache.org/documentation/streams/
Supports
- Kafka Streams as a stream-processing library and its official documentation path
- https://github.com/manuzhang/awesome-streaming
Supports
- Curated discovery of Apache Flink, Apache Beam, and Kafka Streams for ecosystem links