Data Versioning Explained: Why Git Alone Isn't Enough for Data Science

Version control is foundational to modern software development. But in data science, tracking code is only half the problem. The datasets, models, and configurations that drive ML pipelines need versioning too — and Git was never designed for that.

Every data scientist has experienced this scenario: you have a trained model that works perfectly, but you cannot reproduce it because you don't know which version of the training data was used, what preprocessing steps were applied, or which hyperparameters produced the result.

This article explains why Git alone is insufficient for data science workflows, compares five data versioning approaches with real measurements, and shows you how to build reproducible ML pipelines.

The Problem: What Happens When You Put Data in Git

Git tracks files efficiently when they are small and text-based. Code files, configuration, and documentation are perfect Git content. But data science projects also contain:

• Training datasets (megabytes to gigabytes)
• Trained models (megabytes to hundreds of megabytes)
• Processed feature files
• Evaluation results and metrics
• Generated plots and visualizations

When you commit a 200 KB CSV to Git and then modify 5% of the rows and commit again, Git stores both complete versions in the .git/ directory. Our benchmark demonstrated this:

Three versions of roughly 211 KB each means the .git/ directory now contains 633 KB of data — even though the actual differences between versions were tiny. At scale, a 500 MB dataset with 10 versions would produce a 5 GB repository.

Five Approaches to Data Versioning

Let's compare the major approaches, from "don't version data at all" to full pipeline integration:

1. Git Without Data (Code Only)

The simplest approach: don't commit data to Git at all. Store data separately (local disk, S3, shared drive) and keep only code and configuration in Git.

# .gitignore
data/
models/
*.csv
*.parquet
*.pkl
*.h5

Pros: Simple, no extra tools, keeps Git fast.

Cons: No reproducibility — you don't know which data produced which result.

2. Git + Raw Data Files

Commit CSVs, Parquet files, and models directly to Git. This is what many teams do initially.

Pros: Everything is in one place.

Cons: Repository grows exponentially, clones become slow, Git history becomes bloated.

3. Git LFS (Large File Storage)

Git LFS replaces large files with small pointer files in Git and stores the actual content on a separate server (GitHub, GitLab, S3).

# Install Git LFS
git lfs install

# Track large file types
git lfs track "*.csv"
git lfs track "*.parquet"
git lfs track "*.pkl"

Pros: Native Git integration, supported by GitHub/GitLab.

Cons: No pipeline awareness, no dependency tracking, stores full files (no delta), requires LFS server setup.

4. DVC (Data Version Control)

DVC was purpose-built for data science versioning. It stores metadata (small pointer files) in Git and actual data in a separate cache or remote storage (S3, GCS, local disk).

# Initialize DVC in your project
dvc init

# Track a dataset
dvc add data/train.csv

# This creates data/train.csv.dvc (the pointer file)
# The actual data goes to .dvc/cache/

# Push data to remote storage
dvc remote add -d storage s3://my-bucket/dvc-storage
dvc push

The .dvc file that Git tracks looks like this:

# data/train.csv.dvc (what Git actually tracks)
outs:
- md5: edd246b09516c9fdc62eb22f9b5427a3
  size: 518933
  path: train.csv

Git sees a tiny YAML file. DVC knows exactly which version of the data this pointer refers to. The actual 518 KB of data lives in the DVC cache.

5. Metadata-Only Tracking (Lightweight)

For smaller projects, a simple JSON or YAML manifest can track dataset versions without any extra tools:

{
  "dataset_name": "sales_training_data",
  "versions": [
    {
      "version": "1.0",
      "date": "2026-01-15",
      "hash": "9f0288676388fbc8a1b94cccc0268588",
      "size_bytes": 216197,
      "rows": 5000,
      "description": "Initial training dataset"
    },
    {
      "version": "2.0",
      "date": "2026-02-20",
      "hash": "f9fb925d96f68f063746e246ddb47a84",
      "size_bytes": 346154,
      "rows": 8000,
      "description": "Added Q2 data, fixed price outliers"
    }
  ]
}

Pros: No extra dependencies, works with any storage, easy to understand.

Cons: No pipeline integration, no automatic data storage, manual enforcement.

Hash-Based Change Detection

Every data versioning tool relies on file hashing to detect changes. When data changes, the hash changes — even if the file size barely moved:

The modified file is the same size but has a completely different hash — because only 5% of rows were changed. The identical copy has the same hash despite being a separate file. This is how DVC and Git LFS know whether data has actually changed.

Pipeline Lineage: Connecting Data to Results

The most powerful aspect of data versioning is pipeline lineage — tracking how data flows through your ML pipeline and which inputs produced which outputs:

Stage          Input                    Output               Runtime
─────────────  ───────────────────────  ───────────────────  ────────
ingest         raw/sales_2026.csv       clean_sales.csv      12.3s
features       clean_sales.csv          features.parquet     45.7s
train          features.parquet         model_v3.pkl         180.2s
evaluate       features.parquet + model metrics.json         23.1s

DVC pipelines can track each stage's inputs, outputs, parameters, and dependencies. When any input changes, DVC knows which stages need to be re-run:

• Input data changes → Re-run from that stage forward
• Code changes → Re-run affected stages
• Parameters change → Re-run affected stages
• Nothing changed → Skip (cached results)

DVC Workflow in Practice

Here is a practical DVC workflow for an ML project:

# 1. Initialize
git init && dvc init

# 2. Create a DVC-tracked dataset
dvc add data/train.csv
git add data/train.csv.dvc .gitignore
git commit -m "Add training data v1"

# 3. Create a DVC pipeline stage
dvc run -n train \
  -d data/train.csv \
  -d src/train.py \
  -o models/model.pkl \
  -p learning_rate,max_depth \
  python src/train.py

# 4. Push data to remote
dvc push

# 5. Later, reproduce the exact experiment
git clone 
cd 
dvc pull        # Fetch data from remote
dvc repro       # Re-run pipeline if needed

# 6. Check what changed
dvc diff        # Show data differences
dvc metrics show  # Show metrics

When to Use Each Approach

Data Versioning Best Practices

1. Version everything: code (Git), data (DVC), config (Git), models (DVC), pipeline (DVC)
2. Use content hashing: never rely on filenames alone for version identification
3. Pin your environment: version your Python dependencies alongside data (requirements.txt or poetry.lock)
4. Track lineage: record which data produced which model and which metrics
5. Use remote storage: don't rely solely on local disk for data cache
6. Document changes: every data version should have a description of what changed and why
7. Automate validation: run data quality checks as part of your pipeline

The Reproducibility Formula

True reproducibility in ML requires versioning all four components:

Git handles code. DVC handles data and models. YAML/JSON handles configuration. Docker or conda handles the environment. Together, they give you the ability to check out any historical state of your project and reproduce it exactly.

Conclusion

Git is a powerful tool, but it is not a data versioning tool. Data scientists who rely solely on Git for dataset tracking face bloaty repositories, unreproducible experiments, and no pipeline lineage.

Data versioning tools like DVC, Git LFS, and Delta Lake solve these problems by treating data as a first-class citizen alongside code. The right tool depends on your project scale, team size, and infrastructure — but doing nothing is no longer acceptable for serious ML work.

The reproducibility formula is simple: version your code, your data, your configuration, and your environment. Then you can reconstruct any experiment at any point in time.