Version datasets with DVC backed by GCS

Introduction

When a model evaluation reveals a regression, you need to answer a precise question: what exact data did the pipeline consume during the previous run that produced better results? Code versioning with Git handles the transformation logic, but training datasets, embedding snapshots, and evaluation outputs can be gigabytes or terabytes and cannot live in Git directly — losing that tie between a commit and the data it consumed turns "reproduce the run" into guesswork. By the end you'll be able to configure DVC with Google Cloud Storage as the remote backend and version dataset artifacts so every pipeline run carries an immutable reference to the exact data it used.

Key Terminology

• DVC (Data Version Control) — a Git-companion tool that tracks large files via content hashes, keeping lightweight pointer files in Git while storing the actual bytes in a configured remote; lets you snapshot dataset versions alongside code commits.
• GCS remote — a Google Cloud Storage bucket configured as DVC's content-addressable backend (gs://bucket/dvc-cache), where every tracked dataset version is stored and from which pipeline workers fetch on dvc pull.
• .dvc pointer file — a small YAML file committed to Git for each tracked dataset that records the content hash, size, and remote location; checking out a Git ref restores the pointer, and dvc checkout fetches the matching data from GCS.
• Content hash — the MD5 digest DVC computes for tracked files; serves as the immutable dataset version identifier that pipeline logs and Dagster asset metadata reference to prove which exact data a run consumed.

Concepts

DVC works by separating a dataset's identity from its bytes. When you track a file, DVC computes a content hash (MD5 by default) and writes a small .dvc pointer file recording that hash, the file size, and the remote location. The pointer file is committed to Git; the actual bytes are pushed to a content-addressable cache in the GCS remote. Because the hash is derived from the content itself, identical data is stored once and any change produces a new hash — giving each dataset version an immutable identifier.

This split is what makes a Git commit a complete specification of a pipeline run. Checking out a commit restores the .dvc pointers as they existed at that point; dvc checkout (or dvc pull, when the bytes aren't cached locally) then fetches the matching data from GCS. A regression investigation becomes deterministic: check out the prior commit, pull its data, and the pipeline sees exactly the bytes it consumed before.

The content hash also flows outward as lineage. Recording it in pipeline logs and Dagster asset metadata ties every produced artifact to the precise input version it was derived from — so "which data made this model?" is answered by a hash, not a guess. The Code Walkthrough below configures DVC against a GCS remote and tracks a dataset so these guarantees hold across runs.

Code Walkthrough

Building on the previous section, this section demonstrates configuring DVC against a GCS remote and tracking a dataset so the next pipeline run can restore it by Git ref.

Code snippetbash
1# Initialize DVC inside an existing Git repo and point it at GCS
2dvc init
3dvc remote add -d gcs gs://my-pipeline-bucket/dvc-cache
4dvc remote modify gcs credentialpath /secrets/gcs-sa-key.json
5
6# Track a dataset: DVC writes data/train.csv.dvc (pointer) + updates .gitignore
7dvc add data/train.csv
8
9# Commit the pointer to Git and push the bytes to GCS
10git add data/train.csv.dvc data/.gitignore
11git commit -m "Snapshot train.csv for run 2026-05-17"
12dvc push

You'll know it works when gsutil ls gs://my-pipeline-bucket/dvc-cache/ shows content-addressed objects, and git checkout <prior-commit> -- data/train.csv.dvc && dvc checkout restores the earlier dataset version locally.

The bash flow above is what DVCManager automates so pipeline code can version data without shelling out by hand. It wraps the same commands behind four methods.

Code snippetpython
1import subprocess, yaml
2from pathlib import Path
3
4class DVCManager:
5    """Wraps DVC CLI operations for Dagster assets / Argo Workflow containers."""
6
7    def __init__(self, repo_dir: str, gcs_url: str):
8        self.repo_dir = Path(repo_dir)
9        self.gcs_url = gcs_url  # e.g. gs://my-pipeline-bucket/dvc-cache
10
11    def init_and_configure(self) -> None:
12        # --no-scm initialises DVC without requiring a Git repo at this path
13        self._run(["dvc", "init", "--no-scm"])
14        self._run(["dvc", "remote", "add", "-d", "gcs", self.gcs_url])
15        self._run(["dvc", "remote", "modify", "gcs",
16                   "credentialpath", "/secrets/gcs-sa-key.json"])
17
18    def track_dataset(self, path: str) -> str:
19        # dvc add writes <path>.dvc (the MD5 pointer) and updates .gitignore
20        self._run(["dvc", "add", path])
21        dvc_file = f"{path}.dvc"
22        self._run(["git", "add", dvc_file, ".gitignore"])
23        self._run(["git", "commit", "-m", f"Snapshot {path}"])
24        self._run(["dvc", "push"])
25        return self._read_dvc_metadata(dvc_file)  # the content hash
26
27    def checkout_version(self, commit: str, dvc_file: str) -> None:
28        # restore the pointer as it was at <commit>, then sync the workspace from the cache
29        self._run(["git", "checkout", commit, "--", dvc_file])
30        self._run(["dvc", "checkout", dvc_file])
31
32    def _read_dvc_metadata(self, dvc_file: str) -> str:
33        data = yaml.safe_load(Path(dvc_file).read_text())
34        # .dvc files store tracked outputs under the top-level "outs" key
35        return data["outs"][0]["md5"]
36
37    def _run(self, cmd: list[str]) -> None:
38        subprocess.run(cmd, cwd=self.repo_dir, check=True)

track_dataset returns the dataset's content hash so callers can log it as lineage; checkout_version runs git checkout then dvc checkout in that order to restore a historical version (the blobs must already be present via dvc pull).

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Commit the generated .dvc pointer files and .gitignore updates to Git so every code commit pins an exact dataset version.
2. ✓Run dvc push after dvc add so the content-addressed blobs reach the GCS remote before any downstream worker tries to dvc pull.
3. ✓Record the content hash returned by track_dataset as Dagster asset metadata so pipeline runs are traceable to the specific dataset version they consumed.

Don'ts

1. ✗Do not commit large dataset files directly to Git — DVC tracks them in GCS and only the .dvc pointer file belongs in the repository.
2. ✗Do not call checkout_version without first running dvc pull against the configured GCS remote; the .dvc pointers are useless without the cached blobs.
3. ✗Do not share one dvc-cache prefix across unrelated pipelines in the same bucket — isolate caches per pipeline to keep content-addressable storage scoped and auditable.