CI/CD for Machine Learning
CI/CD for machine learning automates how changes to training code, data pipelines, and model-serving code are tested and released. It adds data and model quality gates to familiar software delivery so a new model reaches production only after its inputs, behavior, and serving contract pass defined checks.
itArtificial intelligence and machine learning | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic: CI/CD for Machine Learning
CI/CD for machine learning is a release path for systems whose state is larger than code. Training logic, feature definitions, data expectations, model artifacts, serving images, infrastructure, configuration, and evaluation evidence can all decide whether production is safe. A green build has more paperwork than usual, because the machine learned something and now everyone wants to know from what.
Keep two loops separate. The CI/CD pipeline starts from source change. It tests code, packages components, and deploys pipeline or serving machinery. The ML pipeline starts from approved code and data. It validates data, trains a candidate, evaluates it, and registers an accepted model. Continuous training runs that second loop from a schedule, new data, drift signal, or policy trigger. It is not CI wearing a lab coat.
Fast checks go first because full training is expensive and slow. Unit tests cover feature functions, labels, metrics, and serving adapters. Contract tests check the artifacts passed between steps. A small-data smoke run proves the path executes and emits finite outputs. Only then should the expensive evidence arrive.
The model registry is the handoff point. Approve an immutable model version, not a vague thing named latest. Labels are useful for traffic and workflow, but the audit record needs the exact version underneath. Build once, promote the same artifact, and make rollback point to a known model and serving release.
Machine learning adds a special nuisance: success can be statistical. Metrics need thresholds, tolerances, datasets, and slice checks. Drift is a signal to investigate. It does not prove the current model is bad, and it does not crown the newly trained one king of all predictions.
Start with the Practice Reference to classify loops, gates, and identities. Use the Exercise to design a release path for a feature-logic change. The Cheatsheet is the operational checklist for lineage, promotion, rollback, security, and cost. The goal is not maximum automation. It is a path where every deployed model has a name, evidence, and a safe way back.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
Supports
- CI/CD and continuous-training distinctions and maturity levels
- Source, test, deployment, registry, feature store, metadata, and orchestrator components
- Data and model validation, component tests, deployment compatibility, staged promotion, triggers, and monitoring
- https://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/ci-cd-for-ml
Supports
- CI/CD scope across training data, input pipelines, training and serving code, predictions, and performance
- Configuration as code, infrastructure as code, logging, and troubleshooting roles
- https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/machine-learning-operations-v2
Supports
- End-to-end CI/CD and retraining architectures for classical ML, computer vision, and natural-language workloads
- Azure Machine Learning product placement in managed ML delivery
- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-mlops-azureml?view=azureml-api-2
Supports
- Azure DevOps integration for infrastructure, data preparation, training, deployment, and monitoring
- Model evaluation and promotion through registration
- Environment identity and workload federation guidance
- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-models?view=azureml-api-2
Supports
- Azure Machine Learning model registration, versioning, lifecycle, and registry identities
- https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-projects.html
Supports
- SageMaker Projects CI/CD scope across images, data preparation, features, training, evaluation, deployment, monitoring, and updates
- Amazon SageMaker AI placement in the product landscape
- https://www.tensorflow.org/tfx/guide/understanding_tfx_pipelines
Supports
- TFX artifacts, parameters, components, pipeline graphs, and orchestrators
- Automated preparation, training, evaluation, deployment, and artifact dependency tracking
- https://www.kubeflow.org/docs/components/pipelines/concepts/pipeline/
Supports
- Kubeflow pipeline graphs, component execution, artifact flow, caching, retries, resources, and run history
- Kubeflow placement as a Kubernetes ML pipeline orchestrator
- https://www.kubeflow.org/docs/components/pipelines/interfaces/
Supports
- Kubeflow user interface, SDK, and API roles
- API use from continuous integration and deployment systems
- https://mlflow.org/docs/latest/ml/model-registry/workflow
Supports
- Registered model versions, aliases, tags, source runs, deployment, and cross-environment promotion
- Mutable alias versus immutable model-version behavior
- MLflow placement as a registry and lifecycle layer in an independent CI/CD stack
- https://docs.github.com/en/actions/how-tos/deploy/configure-and-manage-deployments/control-deployments
Supports
- Deployment environments, protection rules, required reviews, secrets, concurrency, triggers, and history
- Protected promotion-job and stage-specific credential guidance
- https://github.com/kelvins/awesome-mlops
Supports
- Discovery of ClearML, CML, and KitOps in the CI/CD for Machine Learning section
- https://clear.ml/docs/latest/docs/
Supports
- ClearML platform scope across compute, development, orchestration, artifacts, and deployment
- ClearML rationale and product placement
- https://github.com/clearml/clearml/blob/master/LICENSE
Supports
- ClearML open-source core licensing behind the open-core classification
- https://clear.ml/pricing/
Supports
- ClearML free and paid managed offerings behind the freemium classification
- https://cml.dev/doc
Supports
- CML use with repository CI for training, evaluation, reports, dataset monitoring, and cloud runners
- CML Awesome Links rationale and product placement
- https://github.com/iterative/cml/blob/master/LICENSE
Supports
- CML Apache-2.0 licensing and free open-source classification
- https://kitops.org/docs/integrations/cicd/
Supports
- ModelKit creation, contents, registry handoff, and use from CI/CD tools
- KitOps Awesome Links rationale and product placement
- https://github.com/kitops-ml/kitops/blob/main/LICENSE
Supports
- KitOps Apache-2.0 licensing and free open-source classification
- https://cloud.google.com/ai-platform/docs/technical-overview
Supports
- Vertex AI training, evaluation, registry, deployment, monitoring, and managed MLOps scope
- Vertex AI product placement
- https://cloud.google.com/vertex-ai/pricing
Supports
- Vertex AI paid managed-service classification
- https://azure.microsoft.com/en-us/pricing/details/machine-learning/
Supports
- Azure Machine Learning paid managed-service classification
- https://aws.amazon.com/sagemaker-ai/pricing/
Supports
- Amazon SageMaker AI paid managed-service classification
- https://www.databricks.com/product/machine-learning
Supports
- Databricks MLflow, feature, model, workflow, deployment, and monitoring product scope
- Databricks Machine Learning product placement
- https://www.databricks.com/product/pricing
Supports
- Databricks paid managed-platform classification
- https://github.com/mlflow/mlflow/blob/master/LICENSE.txt
Supports
- MLflow Apache-2.0 licensing and free open-source classification
- https://github.com/kubeflow/kubeflow/blob/master/LICENSE
Supports
- Kubeflow Apache-2.0 licensing and free open-source classification
- https://docs.gitlab.com/user/project/ml/model_registry/
Supports
- GitLab model versions, metadata, MLflow compatibility, and links from model versions to CI/CD jobs
- GitLab placement in the ML delivery landscape
- https://about.gitlab.com/pricing/
Supports
- GitLab free and paid tiers behind the freemium classification
- https://handbook.gitlab.com/handbook/engineering/open-source/
Supports
- GitLab open-core product model
- https://research.google/pubs/hidden-technical-debt-in-machine-learning-systems/
Supports
- 2015 publication and system-level ML technical-debt concerns
- https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction/
Supports
- 2017 publication and 28 tests and monitoring needs for production ML
- https://blog.kubeflow.org/releases/2020/03/02/kubeflow-1-0-cloud-native-ml-for-everyone.html
Supports
- Kubeflow open sourcing in December 2017
- Kubeflow 1.0 release on March 2, 2020 and its stable production ML core
- https://www.databricks.com/sites/default/files/2023-08/accelerating-the-machine-learning-lifecycle-with-mlflow.pdf
Supports
- MLflow launch in June 2018
- MLflow's original experimentation, reproducibility, and deployment scope
- https://arxiv.org/abs/1903.00278
Supports
- March 2019 publication of an ML-specific continuous integration system
- Statistical model-quality decisions in ML integration
- https://dvc.org/blog/cml-release/
Supports
- CML release on July 7, 2020
- Training, evaluation, review reports, and compute in GitHub and GitLab CI
- https://aws.amazon.com/about-aws/whats-new/2020/12/introducing-amazon-sagemaker-pipelines-first-purpose-built-ci-cd-service-machine-learning/
Supports
- SageMaker Pipelines general availability on December 8, 2020
- Managed workflow orchestration, registry approval, lineage, and MLOps templates
