Don't Panic — Stream Processing

Stream processing is computation that keeps working while records keep arriving. A payment, click, device reading, or database change walks in, and the system has to do something useful before the queue of arrivals politely agrees to end. It will not agree. Batch processing waits for a finished pile of data; streaming makes progress through the pile while somebody is still adding to it.

The useful mental picture is a dataflow: sources provide records, operators change or combine them, and sinks put results somewhere useful. A filter can examine one record and move on. A running total or a join cannot. It needs state, meaning remembered information from earlier records. A key keeps the relevant records with the state that belongs to them, rather than letting every account's history become one large and argumentative heap.

Time is the part that looks innocent until it arrives late. Event time is when something happened. Processing time is when the system noticed. They are only the same in the unusually well-behaved universe where devices never disconnect and networks never delay anything. A window gives an endless stream a finite question, such as a total for one minute. A watermark estimates how far event time has moved, so the system can emit a result without waiting until the heat death of the universe. Late records can still appear, so the policy must say whether to drop, sideline, or correct them.

Failure also joins the meeting. A checkpoint saves operator state together with input positions. After a failure, the processor restores that point and replays what followed it. This is valuable, but it does not make an unrelated external API call happen exactly once. The delivery guarantee has a boundary. If the sink can see a repeated request, its effect needs to tolerate repetition or share the transaction boundary.

Read the Intro when the component flow and tradeoffs need a fuller explanation. Use Slides for the short map from records through recovery. Keep the Cheatsheet nearby when choosing a clock, a window, or a late-data policy. Then run the Practice Reference and Exercise: moving one event across a window boundary is a small experiment, but it makes the whole time model stop pretending to be decorative.