openskills.info
Course catalog
Data and Model Versioning with DVC logoCourse Preview

Data and Model Versioning with DVC

DVC keeps large datasets and model files outside Git while Git tracks small files that identify each exact version. This lets a project restore matching code, data, and models without putting large binaries in the Git repository.

itArtificial intelligence and machine learning

Data and Model Versioning with DVC

DVC is a command-line tool for versioning large data and model files alongside source code. Git stores small DVC metadata files. A local cache and optional remote storage hold the file contents. A Git revision therefore identifies the code and the DVC metadata needed to recover matching artifacts.

This split addresses a practical mismatch. Git is effective for source code and text review, but large or frequently changing binary files make repository history expensive. Object storage handles large files well, but a bucket path alone does not say which dataset trained a model. DVC connects the two systems without replacing either one.

The three layers

A DVC project has three distinct layers:

  1. Git metadata layer. Git versions .dvc files, dvc.yaml, dvc.lock, configuration, code, and other small text files.
  2. Workspace and cache layer. The workspace exposes ordinary paths such as data/train/ or models/classifier.pkl. DVC stores content-addressed objects in its cache and links or copies the selected objects into those paths.
  3. Remote storage layer. A DVC remote stores cached objects for sharing, backup, and use on another machine. Supported remote types include local filesystems and common object-storage services.

The metadata file is not the data. It records a path, a content hash, and related information. The hash identifies the cached object. Git records changes to that metadata file, so each Git revision can select a different artifact version.

The versioning flow

The core flow begins inside a Git repository. dvc init creates DVC project files. dvc add data/train hashes the target, places its content in the cache, writes data/train.dvc, and arranges for the original path to stay out of Git. You commit the .dvc file and its generated .gitignore entry to Git.

Continue the course

This section is part of the paid course.

See pricing to subscribe, or log in if you already have access.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Sources