openskills.info
Course catalog
Course Preview

Streaming Feature Engineering

Streaming feature engineering turns continuously arriving events into current model inputs. It keeps per-entity state, applies time-aware windows, and makes the resulting values available for both live predictions and historically correct training data.

itArtificial intelligence and machine learning

Streaming Feature Engineering

Streaming feature engineering converts an unbounded flow of events into model inputs that update as new events arrive. A transaction count for the last ten minutes, time since an account's previous login, or current session value can all be streaming features. The computation runs continuously because a live prediction cannot wait for the next batch job.

The central difficulty is not the arithmetic. It is preserving meaning across time. Each event has an entity key, an event timestamp, and a payload. The pipeline must decide which events belong together, which time interval contributes to each value, how long to wait for late events, and how corrections reach training and serving systems.

Architecture and data flow

A typical path has six stages:

  1. An event source records facts such as views, payments, or sensor readings.
  2. A stream processor validates each event, assigns or reads its event timestamp, and partitions it by entity key.
  3. A stateful transformation retains the information needed for windows, joins, counters, or recency calculations.
  4. The processor emits a feature value with its entity key and feature timestamp.
  5. An online store keeps a serving-ready value for low-latency inference. An offline history keeps timestamped values for training and replay.
  6. A model service retrieves the current feature vector for an entity. A training job uses a point-in-time join to reconstruct values as they existed at each labeled event.

The stream processor and feature store solve different parts of this path. Apache Flink, Apache Spark Structured Streaming, Apache Kafka Streams, and similar engines compute stateful transformations. Feature platforms such as Feast catalog feature definitions and connect historical retrieval with online serving. Some managed platforms combine both responsibilities.

Continue the course

This section is part of the paid course.

See pricing to subscribe, or log in if you already have access.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Sources