Train, Validation and Test Sets Explained Properly

You trained a model. It scored 95% accuracy. You deployed it. Real-world performance: 65%.

The problem was not your model. The problem was your evaluation.

Most beginners split data into train and test, tune on the test set, and deploy. This creates a dangerously optimistic estimate of performance.

The correct approach uses three separate datasets. This article explains why, shows you how, and demonstrates when cross-validation changes the workflow.

Why Three Sets, Not Two?

Think of building a model like studying for an exam:

If you study the practice exams (validation) and then take the same practice exams as your final exam (test), your score does not reflect true knowledge.

The Problem with Two Sets

When you only have train and test:

1. You train on the training set
2. You evaluate on the test set
3. You adjust hyperparameters to improve test performance
4. Now your test set has influenced model selection
5. Your test score is no longer unbiased

# ❌ WRONG: Tuning on test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Trying different hyperparameters
for n_estimators in [10, 50, 100, 200]:
    model = RandomForestClassifier(n_estimators=n_estimators)
    model.fit(X_train, y_train)
    score = model.score(X_test, y_test)  # ← Using test for decisions!
    print(f"n_estimators={n_estimators}, accuracy={score:.3f}")

The Correct Three-Way Split

# ✅ CORRECT: Three-way split
from sklearn.model_selection import train_test_split

# First split: 80% train+val, 20% test
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Second split: 75% train, 25% validation (of the 80%)
X_train, X_val, y_train, y_val = train_test_split(
    X_trainval, y_trainval, test_size=0.25, random_state=42
)

# Result: 60% train, 20% validation, 20% test

The Complete Workflow

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Step 1: Split data
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_trainval, y_trainval, test_size=0.25, random_state=42
)

# Step 2: Train on training set
model = RandomForestClassifier(n_estimators=100)
model.fit(X_train, y_train)

# Step 3: Tune on validation set (NOT test!)
val_score = accuracy_score(y_val, model.predict(X_val))
print(f"Validation accuracy: {val_score:.3f}")

# Step 4: Final evaluation on test set (ONLY ONCE)
test_score = accuracy_score(y_test, model.predict(X_test))
print(f"Test accuracy: {test_score:.3f}")

When Cross-Validation Changes the Workflow

Cross-validation replaces the validation set, not the test set. You still need a held-out test set for final evaluation.

How K-Fold Cross-Validation Works

1. Split training data into K folds (e.g., K=5)
2. For each fold: train on K-1 folds, validate on 1 fold
3. Average the K validation scores
4. Test set remains untouched until final evaluation

from sklearn.model_selection import cross_val_score, train_test_split

# Step 1: Split off test set (NEVER use during training)
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Step 2: Cross-validation on train+val
model = RandomForestClassifier(n_estimators=100)
cv_scores = cross_val_score(model, X_trainval, y_trainval, cv=5)

print(f"CV scores: {cv_scores}")
print(f"Mean CV accuracy: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")

# Step 3: Train final model on ALL train+val data
model.fit(X_trainval, y_trainval)

# Step 4: Final evaluation on test set (ONLY ONCE)
test_score = model.score(X_test, y_test)
print(f"Test accuracy: {test_score:.3f}")

Standard Split vs Cross-Validation

Cross-Validation WITH a Test Set

This is the correct workflow when using cross-validation:

from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier

# Step 1: Split off test set
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Step 2: Create pipeline (prevents preprocessing leakage)
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', RandomForestClassifier(n_estimators=100))
])

# Step 3: Cross-validation for model selection
cv_scores = cross_val_score(pipeline, X_trainval, y_trainval, cv=5)
print(f"CV accuracy: {cv_scores.mean():.3f}")

# Step 4: Train final model on ALL train+val data
pipeline.fit(X_trainval, y_trainval)

# Step 5: Final evaluation on test set
test_score = pipeline.score(X_test, y_test)
print(f"Test accuracy: {test_score:.3f}")

Stratified Splitting

For classification problems, use stratified splitting to maintain class proportions:

# Stratified split maintains class balance
X_train, X_test, y_train, y_test = train_test_split(
    X, y, 
    test_size=0.2, 
    stratify=y,  # Maintains class proportions
    random_state=42
)

# Verify class distribution
print("Original:", dict(pd.Series(y).value_counts(normalize=True)))
print("Train:", dict(pd.Series(y_train).value_counts(normalize=True)))
print("Test:", dict(pd.Series(y_test).value_counts(normalize=True)))

Common Mistakes

Dataset Size Guidelines

# Large dataset: simpler split
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.3, random_state=42
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, random_state=42
)
# 70% train, 15% validation, 15% test

Putting It All Together

from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

# === STEP 1: Split off test set ===
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

# === STEP 2: Cross-validation for model selection ===
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', RandomForestClassifier(n_estimators=100, random_state=42))
])

cv_scores = cross_val_score(pipeline, X_trainval, y_trainval, cv=5)
print(f"CV Accuracy: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")

# === STEP 3: Train final model ===
pipeline.fit(X_trainval, y_trainval)

# === STEP 4: Final test evaluation (ONCE) ===
y_pred = pipeline.predict(X_test)
print(classification_report(y_test, y_pred))

Try It Yourself

• Data Leakage in ML — 10 mistakes that destroy your model
• Python Data Science Optimization — Efficient ML workflows
• Python Docker Workspace — Reproducible environments

Further Reading

• Scikit-learn: Cross-validation Guide
• Scikit-learn: Common Pitfalls
• Scikit-learn: train_test_split
• BestWordz: Data Leakage in ML

Conclusion

The three-way split is not optional. Training, validation, and test sets serve different purposes:

• Training — The model learns from this data
• Validation — You make decisions about hyperparameters and model selection
• Test — The final, unbiased estimate of performance

Cross-validation improves the validation step by averaging over multiple splits. But it does not replace the test set. You still need a held-out test set for final evaluation.

The key principles are simple:

1. Split your data before any preprocessing
2. Use cross-validation or a validation set for model selection
3. Evaluate on the test set only once
4. Never let test data influence any training decisions

Your model's test score is only meaningful if the test set was truly held out.