Data Leakage in Machine Learning: 10 Mistakes That Destroy Your Model

Your model achieves 99% accuracy on the test set. You deploy it. Real-world accuracy: 60%.

What happened? Most likely: data leakage.

Data leakage is one of the most dangerous problems in machine learning because it is invisible. Your metrics look great. Your model learns patterns that exist in your training data but will never exist in production.

This article explains the 10 most common data leakage mistakes, shows Python examples of each, and teaches you how to prevent them.

What is Data Leakage?

Data leakage happens when information from outside the training dataset is used to create the model. The model learns shortcuts that work on your test set but fail in the real world.

IBM defines it as: "Information that would not be available at prediction time leaking into the training process."

The four main types:

The 10 Data Leakage Mistakes

Mistake #1: Splitting After Preprocessing

This is the most common mistake. Beginners preprocess the entire dataset, then split into train/test.

# ❌ WRONG: Preprocess before split
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Combines train and test information
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # Uses ALL data

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.2
)

# ✅ CORRECT: Split first, then preprocess
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2
)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Only training data
X_test_scaled = scaler.transform(X_test)  # Don't fit!

Mistake #2: Target Leakage Through Features

Target leakage occurs when a feature contains information that directly or indirectly reveals the target variable.

# ❌ WRONG: Feature contains future target info
import pandas as pd

# Predicting if patient has diabetes
df = pd.DataFrame({
    'age': [45, 52, 38, 61],
    'weight': [80, 95, 72, 88],
    'hba1c_level': [8.2, 9.1, 5.4, 7.8],  # This IS the diagnosis!
    'has_diabetes': [1, 1, 0, 1]
})

# hba1c_level directly indicates diabetes
# Model learns this correlation, not real prediction

# ✅ CORRECT: Remove features that reveal the target
df_clean = df.drop(columns=['hba1c_level'])

# Use features that would be available BEFORE diagnosis
# age, weight, family_history, BMI, etc.

Mistake #3: Data Leakage in Cross-Validation

Cross-validation should simulate production. If preprocessing leaks across folds, your CV scores are unreliable.

# ❌ WRONG: Preprocess before cross-validation
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline

# Leakage: scaler sees all folds
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

scores = cross_val_score(model, X_scaled, y, cv=5)

# ✅ CORRECT: Use a Pipeline
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Pipeline preprocesses within each fold
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', RandomForestClassifier())
])

scores = cross_val_score(pipeline, X, y, cv=5)

Mistake #4: Temporal Leakage with Random Split

Time-series data must be split chronologically, not randomly.

# ❌ WRONG: Random split on time-ordered data
df = pd.DataFrame({
    'date': pd.date_range('2024-01-01', periods=1000),
    'feature_1': range(1000),
    'target': [0,1]*500
})

# Random split mixes past and future
X_train, X_test, y_train, y_test = train_test_split(
    df[['feature_1']], df['target'], test_size=0.2
)

# ✅ CORRECT: Time-based split
train_size = int(len(df) * 0.8)

# Train on past, test on future
X_train = df[['feature_1']].iloc[:train_size]
X_test = df[['feature_1']].iloc[train_size:]
y_train = df['target'].iloc[:train_size]
y_test = df['target'].iloc[train_size:]

Mistake #5: Including ID Columns

Unique identifiers can cause leakage if they correlate with the target.

# ❌ WRONG: ID column leaks information
df = pd.DataFrame({
    'patient_id': [1001, 1002, 1003, 1004],
    'feature_1': [0.5, 0.8, 0.3, 0.9],
    'target': [1, 1, 0, 1]
})

# patient_id might accidentally correlate with target

# ✅ CORRECT: Remove ID columns
X = df.drop(columns=['patient_id', 'target'])

Mistake #6: Leakage in Feature Engineering

Feature engineering must use only training data statistics.

# ❌ WRONG: Compute mean using all data
mean_age = df['age'].mean()  # Includes test data!
df['age_normalized'] = df['age'] / mean_age

# ✅ CORRECT: Compute statistics from training data only
mean_age = X_train['age'].mean()  # Only training data
X_train['age_normalized'] = X_train['age'] / mean_age
X_test['age_normalized'] = X_test['age'] / mean_age

Mistake #7: Data Duplication Across Splits

Duplicate rows in both train and test sets create leakage.

# ❌ WRONG: Duplicates across splits
df = pd.DataFrame({
    'feature': [1, 2, 3, 1, 2, 4],  # Rows 0,3 and 1,4 are duplicates
    'target': [0, 0, 1, 0, 0, 1]
})

X_train, X_test, y_train, y_test = train_test_split(
    df[['feature']], df['target'], test_size=0.3
)

# ✅ CORRECT: Remove duplicates before splitting
df_deduped = df.drop_duplicates()
X_train, X_test, y_train, y_test = train_test_split(
    df_deduped[['feature']], df_deduped['target'], test_size=0.3
)

Mistake #8: Target Encoding Before Split

Target encoding uses the target variable to create features. Doing this before splitting leaks the target into features.

# ❌ WRONG: Target encoding before split
from category_encoders import TargetEncoder

# Uses target information from all data
encoder = TargetEncoder()
X_encoded = encoder.fit_transform(X, y)

# ✅ CORRECT: Target encoding within CV folds
from sklearn.model_selection import KFold

kf = KFold(n_splits=5, shuffle=False)
for train_idx, val_idx in kf.split(X):
    X_train_fold = X.iloc[train_idx]
    y_train_fold = y.iloc[train_idx]
    
    encoder = TargetEncoder()
    X_train_encoded = encoder.fit_transform(X_train_fold, y_train_fold)
    X_val_encoded = encoder.transform(X.iloc[val_idx])

Mistake #9: Leakage Through Test Set Exploration

Peeking at test data influences your decisions about preprocessing, feature selection, or model selection.

# ❌ WRONG: Exploring test set before final evaluation
X_train, X_test, y_train, y_test = train_test_split(X, y)

# Looking at test distribution
print(X_test.describe())  # This influences your decisions!

# Adjusting preprocessing based on test statistics
# Using test set for hyperparameter tuning

# ✅ CORRECT: Use validation set for exploration
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5)

# Explore validation set, not test set
print(X_val.describe())  # OK to explore

# Final test set is used ONLY once at the very end

Mistake #10: Aggregate Statistics Before Split

Computing group statistics using the full dataset before splitting creates leakage.

# ❌ WRONG: Group statistics using all data
df['dept_avg'] = df.groupby('department')['target'].transform('mean')

# This directly leaks the target into a feature!

# ✅ CORRECT: Compute group statistics from training data only
dept_means = X_train.join(y_train).groupby('department')['target'].mean()

X_train['dept_avg'] = X_train['department'].map(dept_means)
X_test['dept_avg'] = X_test['department'].map(dept_means)

How to Detect Data Leakage

Prevention Checklist

Try It Yourself

Ready to build robust ML pipelines?

• Python Data Science Optimization — Efficient ML workflows on standard hardware
• Python Docker Workspace — Reproducible ML environments
• Local AI Model Selection — Choose models for your hardware

Further Reading

• IBM: What is Data Leakage in Machine Learning?
• Machine Learning Mastery: 3 Subtle Ways Data Leakage Can Ruin Your Models
• Scikit-learn: Common Pitfalls
• BestWordz: Python Data Science Optimization

Conclusion

Data leakage is silent. It gives you great metrics and terrible production performance. The only way to catch it is to understand the patterns.

The fundamental principle is simple: never let information from outside the training set influence your model.

Split your data first. Preprocess on training data only. Use Pipeline for cross-validation. Keep your test set locked away until final evaluation.

The 10 mistakes in this article are common, but they are all preventable. Add the prevention checklist to your ML workflow and check it before every model deployment.

Your model's test accuracy means nothing if it leaked data. Build systems that prevent leakage by design.