Course Preview

Streaming Feature Engineering

Streaming feature engineering turns continuously arriving events into current model inputs. It keeps per-entity state, applies time-aware windows, and makes the resulting values available for both live predictions and historically correct training data.

itArtificial intelligence and machine learning | OpenSkills.info

Course pathWalk it in order

Look it upDip in anytime

Go furtherLeaves this page

Don’t Panic — Streaming Feature Engineering

Streaming feature engineering is the machinery that turns arriving events into model inputs before the next batch job has even found its shoes. A purchase, login, or sensor reading arrives with an entity key and an event timestamp. The pipeline keeps the relevant state for that entity, then emits a timestamped value such as a recent purchase total. The number is not interesting by itself. Its meaning is the time interval and the events allowed into it, which is considerably less glamorous but much more useful.

The first awkward fact is that the stream has two clocks. Event time says when something happened; processing time says when a computer happened to notice. Those are often different, because networks delay, retries recur, and mobile devices have their own ideas about punctuality. A watermark is the processor estimate of how far event time has progressed. It lets a window emit instead of waiting indefinitely, but it is an estimate, not a polite notice that all older events have left the building.

The second important idea is the feature contract. It names the entity key, window, timestamps, missing-value behavior, lateness policy, retention, and correction rule. That contract travels down two paths. Online serving asks for the latest eligible value by key. Training uses a point-in-time join to retrieve the value available at a label time, never a later one. A shared feature name does not make these paths agree; matching rules do.

There is a practical catch. A fast lookup can return a stale feature, and an exactly-once state update does not make every external write exactly once. Watch feature age, watermark lag, state size, checkpoint duration, and parity differences, because each points at a different piece of the machine. When batch freshness meets the prediction deadline, batch often has fewer ways to become an expensive philosophical discussion.

Read the Intro for the complete architecture and the choices behind it. Use Slides when the two clocks, windows, and retrieval paths need a map. Keep the Cheatsheet nearby when defining a feature contract or investigating a mismatch. The practice reference and exercise turn the same ideas into a bounded replay, where time has nowhere to hide.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Sources