Kubeflow Pipelines

Kubeflow Pipelines is a system for defining and running repeatable machine learning workflows on Kubernetes. It turns a graph of containerized steps into tracked runs, coordinates their inputs and outputs, and records the artifacts each run produces.

itArtificial intelligence and machine learning

Kubeflow Pipelines

Kubeflow Pipelines, commonly shortened to KFP, orchestrates machine learning workflows on Kubernetes. A pipeline describes a directed graph of components. Each component packages one unit of work, such as preparing a dataset, training a model, or evaluating metrics. The KFP backend turns that graph into Kubernetes workloads, coordinates their dependencies, and records what happened during each run.

KFP separates a workflow definition from a workflow execution. Python code written with the KFP domain-specific language defines components, their typed inputs and outputs, and the connections between them. The compiler converts that definition into an intermediate representation called IR YAML. A KFP backend accepts the IR YAML and creates a run. This boundary makes the compiled package an inspectable deployment artifact rather than an instruction to execute the pipeline during Python import.

The execution path

The normal path has six stages:

Component functions or container specifications define isolated tasks.
A pipeline function connects task outputs to downstream inputs and adds control flow.
The KFP compiler checks the graph and emits IR YAML.
A user, client, API, or recurring-run configuration submits that package with runtime parameters.
The backend schedules task containers as Kubernetes Pods and follows the graph's dependencies.
The UI, API, artifact store, and metadata store expose status, logs, outputs, and lineage.

The decorated pipeline function resembles ordinary Python, but it constructs topology. Calling a component inside that function creates a task object. It does not run the component body. Data connections establish ordering: when one task consumes another task's output, the producer must finish before the consumer starts. Tasks with no dependency can run in parallel.

Kubeflow Pipelines

The execution path

Relevant careers