MLflow for Experiment Tracking and Model Registry
MLflow is an open-source platform that records the details of every machine learning training run — its parameters, metrics, and output files — and keeps a versioned catalog of the models built from those runs, so teams can compare experiments and know exactly which model is running in production.
itArtificial intelligence and machine learning | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic — MLflow for Experiment Tracking and Model Registry
MLflow is the ledger for machine-learning work that otherwise becomes a
collection of hopeful filenames and an alarming amount of memory. Training a
model does not produce one fact. It produces code, hyperparameters, training
data, library versions, metrics, plots, and a model artifact. Leaving those in
separate places is how final_model_really_final.pkl acquires siblings.
The useful shape is experiment, run, then logged model. An experiment groups one line of inquiry. A run is one execution of the training code. A logged model is the artifact from that execution, with a link back to its run. That last link is the point: a model can be traced to the parameters, metrics, and dataset reference that produced it instead of being treated as a mysterious object that appeared during a notebook session.
The slightly sneaky part is that MLflow has two storage jobs. The backend store keeps small, queryable run facts such as parameters, metrics, tags, and status. The artifact store keeps the bulky things: models, plots, and reports. A tracking server puts an HTTP front door in front of both, so a team can see the same experiments. A local directory is fine for one person. It is not a tiny distributed systems miracle in a hat.
When a run produces something worth keeping, the Model Registry gives it a
name and sequential versions. The surprising bit is that a deployment does not
need to chase the newest number. An alias such as champion is a movable
pointer to a chosen version. Consumers ask for the alias; promotion changes the
pointer. This replaces the older fixed stages, which were too rigid for
workflows where several labels or environments must coexist.
Neither logging nor registration decides whether a model is good. MLflow records what the code computes. It does not inspect whether the metric is the right one, whether the data was sound, or whether the result should reach production. That judgment remains inconveniently biological. The record makes it possible to audit the judgment afterwards, which is considerably better than recovering it from a chat message and a half-remembered run ID.
Start with the Intro for the architecture and vocabulary. Use the Slides when
the relationship between stores, server, runs, flavors, and aliases needs a
map. The Cheatsheet is the compact lookup for URIs and operational signals.
Then use the Exercise to run two local training attempts, compare their
evidence, and make one selected model answer to champion. Once that feels
ordinary, a shared tracking server is no longer mysterious machinery. It is
the filing cabinet that prevents an experiment from becoming folklore.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://mlflow.org/docs/latest/
Supports
- MLflow is an open-source AI platform / Linux Foundation project
- MLflow's two main functional areas (LLMs & Agents, and traditional Machine Learning: tracking, model packaging, registry, deployment)
- Availability of both open-source MLflow and managed MLflow on Databricks
- https://mlflow.org/docs/latest/ml/tracking/
Supports
- Experiment, run, and logged model hierarchy
- What a run logs (parameters, metrics, artifacts, tags, log_input for datasets)
- Backend store vs. artifact store distinction
- Tracking server as an optional standalone HTTP process
- mlflow ui vs. mlflow server distinction
- MLFLOW_TRACKING_URI as the client-side pointer to a tracking server
- https://mlflow.org/docs/latest/ml/tracking/autolog
Supports
- mlflow.autolog() behavior and what it captures
- Per-library autologging coverage and limitations (scikit-learn, XGBoost, LightGBM, Keras/TensorFlow, PyTorch Lightning, PySpark ML, Statsmodels, Paddle)
- https://mlflow.org/docs/latest/ml/model-registry/
Supports
- Registered models, model versions, and automatic version numbering
- models:/<name>/<version> URI format
- Model lineage linking a version back to its producing run
- Tags and aliases as registry metadata mechanisms
- https://mlflow.org/docs/latest/ml/model-registry/workflow
Supports
- Aliases (e.g. champion, challenger) as mutable named references to a model version
- Stage deprecation since MLflow 2.9 and its rationale
- Environment-scoped registered models (dev/staging/prod) as the current isolation pattern
- copy_model_version() for promoting a version across environment-scoped models
- models:/<name>@<alias> URI format
- https://mlflow.org/docs/latest/ml/model/
Supports
- MLmodel file structure and purpose
- Model flavors concept, including python_function as the universal flavor
- Model signature and input example
- serving_input_example.json generation
- Bundled dependency files (requirements.txt, conda.yaml, python_env.yaml)
- https://mlflow.org/docs/latest/ml/projects/
Supports
- MLproject file structure (name, environment, entry points)
- Environment types (virtualenv, Conda, Docker, system) and their tradeoffs
- mlflow run command and execution against local or remote Git projects
- https://mlflow.org/docs/latest/ml/deployment/
Supports
- mlflow models serve for local REST serving
- mlflow models build-docker for containerized deployment
- Managed deployment targets (SageMaker, Azure ML, Databricks Model Serving)
- Community-supported Kubernetes targets (KServe, Ray Serve)
- python_function flavor as the common interface across deployment targets
- https://mlflow.org/docs/latest/ml/mlflow-3/
Supports
- LoggedModel as a first-class entity introduced in MLflow 3
- Model artifacts moved from run-relative paths to experiments/<id>/models/<id>/artifacts/
- Impact on mlflow.get_artifact_uri("model") and other run-relative model access code
- log_model callable without mlflow.start_run() context in MLflow 3
- https://mlflow.org/releases/
Supports
- Current MLflow release version (3.15.1) and release date used for course freshness snapshot
- https://optuna-integration.readthedocs.io/en/stable/reference/generated/optuna_integration.MLflowCallback.html
Supports
- Optuna's MLflowCallback logging trial parameters and scores into an MLflow experiment (Awesome Links rationale)
- https://docs.bentoml.com/en/latest/reference/frameworks/mlflow.html
Supports
- BentoML's dedicated MLflow framework integration for importing and serving MLflow-format models (Awesome Links rationale)
- https://github.com/SeldonIO/MLServer/tree/master/runtimes/mlflow
Supports
- MLServer's dedicated MLflow runtime and V2 Inference Protocol signature conversion (Awesome Links rationale)
- https://doc.dvc.org/
Supports
- DVC's purpose: versioning datasets and models alongside code, reproducible pipelines (Awesome Links rationale)
- https://wandb.ai/site
Supports
- Weights & Biases as an experiment tracking and ML platform covering tracking, registry, and sweeps (Awesome Links rationale)
- https://www.kubeflow.org/
Supports
- Kubeflow as a Kubernetes-native AI platform
- Kubeflow Hub as a model registry component (Awesome Links rationale)
- https://mlflow.org/releases/archive
Supports
- Release dates for MLflow 0.1.0, 2.0, 2.9, and 3.0 in the Timeline
- https://github.com/mlflow/mlflow/blob/master/changelogs/v1.x.md
Supports
- MLflow 1.0 stable APIs and step-based metric visualization
- MLflow 1.4 Model Registry beta
- MLflow 1.10 registry client APIs
- MLflow 1.11 scikit-learn autologging
- https://github.com/mlflow/mlflow/issues/10336
Supports
- Model-version aliases and tags replacing fixed registry stages
- Stage deprecation beginning with MLflow 2.9
- https://mlflow.org/blog/archive
Supports
- Introduction of MLflow Tracing in June 2024
- https://mlflow.org/docs/latest/ml/model-registry/tutorial
Supports
- Local tracking server and model registry tutorial workflow
- Logging a model, registering it, and setting a model-version alias
- SQLite and local filesystem structure for a learning setup
- https://www.comet.com/docs/v2/guides/experiment-management/quickstart/
Supports
- Comet experiment tracking as a centralized record of training runs
- https://www.comet.com/docs/v2/guides/model-registry/using-model-registry/
Supports
- Comet model registry workflow from an experiment model to registered versions
- https://docs.develop.azure.neptune.ai/3.4.14/about/
Supports
- Neptune experiment tracking, comparison, visualization, and monitoring features
- https://docs.gitlab.com/user/project/ml/model_registry/
Supports
- GitLab Model Registry and MLflow-compatible model version management
- https://www.evidentlyai.com/
Supports
- Evidently AI's purpose: monitoring data drift and predictive/AI system performance after deployment (Awesome Links rationale)