Course Preview

Kubeflow Pipelines

Kubeflow Pipelines is a system for defining and running repeatable machine learning workflows on Kubernetes. It turns a graph of containerized steps into tracked runs, coordinates their inputs and outputs, and records the artifacts each run produces.

itArtificial intelligence and machine learning | OpenSkills.info

Course pathWalk it in order

Look it upDip in anytime

Go furtherLeaves this page

Don't Panic — Kubeflow Pipelines

Kubeflow Pipelines turns Python component definitions into Kubernetes workloads, one container per step, with run history, artifact tracking, and lineage to prove what happened.

Before KFP, teams ran ML workflows with ad-hoc scripts, cron jobs, and tribal knowledge about which step to run next. Results were unreproducible, failures were opaque, and nobody could tell you which dataset trained the model currently in production.

The translation chain is the idea everything else hangs off: Python code becomes a graph, the graph compiles to IR YAML, and a backend turns that YAML into Kubernetes Pods. Each component lives in its own container. Components never share memory. They exchange data through parameters (small JSON values) and artifacts (datasets, models, metrics stored at a URI with metadata). Once that boundary clicks, the rest of KFP is detail.

The thing that surprises most people: calling your pipeline function in Python does not run the pipeline. It constructs a graph topology. The compiler checks that topology and produces a package. You submit the package to a backend, which creates a run. The Python import is not the execution. The compiled YAML is the deployment artifact, and you can inspect every task, every dependency, and every typed connection before anything touches a cluster.

The second surprise lives in the gap between "compiles" and "runs." The compiler validates types, connections, and graph shape. It does not check whether your container image exists, your PersistentVolumeClaim is bound, your ConfigMap lives in the target namespace, or your mount path actually points somewhere real. Kubernetes checks those at Pod startup. A task that compiled clean can sit in Pending while the scheduler hunts for GPU capacity, or fail because a volume path has a stray character.

When a run fails, follow the translation chain in reverse. Inspect the run graph to find the failed task. Read its Kubernetes Pod logs. Check the Pod state: Pending points to scheduling, quota, image pull, or volume constraints. Succeeded-but-artifact-missing points to the artifact path, object storage, or permissions. Caching hid a changing input is its own category — the cache key matched, the upstream data did not.

KFP covers workflow authoring, orchestration, run tracking, and lineage. It does not replace source control, continuous integration, a feature store, a model registry, serving, or monitoring. The Landscape tab shows what sits around it. The Reference tab gives you the official documentation path from basics through migration. The Field Notes tab carries the judgment: what teams get wrong and why the standard advice has costs nobody states.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Sources