Model Evaluation Beyond Accuracy

A fraud detection model achieves 99.5% accuracy. Management is thrilled.

The model predicts "not fraud" for every transaction. It catches zero fraud cases.

Accuracy told you the model was great. The confusion matrix tells you the truth.

This article explains why accuracy is insufficient, introduces the metrics that matter, and shows Python examples for each.

The Confusion Matrix

Every classification metric starts with the confusion matrix.

import numpy as np
from sklearn.metrics import confusion_matrix, classification_report

# Actual and predicted labels
y_true = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
y_pred = [0, 0, 0, 1, 0, 1, 1, 1, 1, 0]

# Confusion matrix
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()

print(f"True Negatives:  {tn}")
print(f"False Positives: {fp}")
print(f"False Negatives: {fn}")
print(f"True Positives:  {tp}")

# Output:
# True Negatives:  3
# False Positives: 1
# False Negatives: 2
# True Positives:  4

Confusion Matrix Layout

Precision

Precision measures how many predicted positives are actually positive.

from sklearn.metrics import precision_score

precision = precision_score(y_true, y_pred)
print(f"Precision: {precision:.3f}")
# Precision: 0.800

# Formula:
# precision = tp / (tp + fp)
# precision = 4 / (4 + 1) = 0.800

When Precision Matters

• Spam detection — Don't mark legitimate emails as spam
• Content moderation — Don't remove valid content
• Recommendation systems — Don't recommend irrelevant items

Recall

Recall measures how many actual positives are correctly identified.

from sklearn.metrics import recall_score

recall = recall_score(y_true, y_pred)
print(f"Recall: {recall:.3f}")
# Recall: 0.667

# Formula:
# recall = tp / (tp + fn)
# recall = 4 / (4 + 2) = 0.667

When Recall Matters

• Cancer detection — Don't miss actual cancers
• Fraud detection — Don't miss fraudulent transactions
• Safety systems — Don't miss dangerous situations

F1 Score

F1 Score is the harmonic mean of precision and recall.

from sklearn.metrics import f1_score

f1 = f1_score(y_true, y_pred)
print(f"F1 Score: {f1:.3f}")
# F1 Score: 0.727

# Formula:
# f1 = 2 * (precision * recall) / (precision + recall)
# f1 = 2 * (0.800 * 0.667) / (0.800 + 0.667) = 0.727

F-beta Score

from sklearn.metrics import fbeta_score

# F2: Weight recall higher than precision
f2 = fbeta_score(y_true, y_pred, beta=2)
print(f"F2 Score: {f2:.3f}")

# F0.5: Weight precision higher than recall
f05 = fbeta_score(y_true, y_pred, beta=0.5)
print(f"F0.5 Score: {f05:.3f}")

ROC-AUC

ROC-AUC measures the model's ability to distinguish between classes across all thresholds.

from sklearn.metrics import roc_auc_score, roc_curve
import matplotlib.pyplot as plt

# Predicted probabilities
y_proba = [0.1, 0.2, 0.3, 0.6, 0.4, 0.7, 0.8, 0.9, 0.85, 0.35]

# ROC-AUC score
roc_auc = roc_auc_score(y_true, y_proba)
print(f"ROC-AUC: {roc_auc:.3f}")

# Plot ROC curve
fpr, tpr, thresholds = roc_curve(y_true, y_proba)

plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, label=f'ROC Curve (AUC = {roc_auc:.3f})')
plt.plot([0, 1], [0, 1], 'k--', label='Random Classifier')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend()
plt.show()

Interpreting ROC-AUC

PR-AUC

PR-AUC (Precision-Recall AUC) is more informative than ROC-AUC for imbalanced datasets.

from sklearn.metrics import average_precision_score, precision_recall_curve

# PR-AUC score
pr_auc = average_precision_score(y_true, y_proba)
print(f"PR-AUC: {pr_auc:.3f}")

# Plot Precision-Recall curve
precision_vals, recall_vals, _ = precision_recall_curve(y_true, y_proba)

plt.figure(figsize=(8, 6))
plt.plot(recall_vals, precision_vals, label=f'PR Curve (AUC = {pr_auc:.3f})')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve')
plt.legend()
plt.show()

ROC-AUC vs PR-AUC

Calibration

Calibration measures whether predicted probabilities match actual outcomes.

from sklearn.calibration import calibration_curve, CalibratedClassifierCV
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification
import matplotlib.pyplot as plt

# Generate imbalanced dataset
X, y = make_classification(
    n_samples=1000, n_features=10,
    n_informative=5, n_redundant=2,
    weights=[0.9, 0.1],  # 90% negative, 10% positive
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# Train model
model = LogisticRegression(random_state=42)
model.fit(X_train, y_train)

# Get predicted probabilities
y_proba = model.predict_proba(X_test)[:, 1]

# Calibration curve
fraction_of_positives, mean_predicted_value = calibration_curve(
    y_test, y_proba, n_bins=10
)

# Plot
plt.figure(figsize=(8, 6))
plt.plot(mean_predicted_value, fraction_of_positives, 's-', label='Model')
plt.plot([0, 1], [0, 1], 'k--', label='Perfectly calibrated')
plt.xlabel('Mean Predicted Probability')
plt.ylabel('Fraction of Positives')
plt.title('Calibration Curve')
plt.legend()
plt.show()

Calibration Metrics

from sklearn.metrics import brier_score_loss

# Brier score (lower is better)
brier = brier_score_loss(y_test, y_proba)
print(f"Brier Score: {brier:.4f}")
# Brier Score: 0.0654

# Expected Calibration Error (ECE)
def expected_calibration_error(y_true, y_prob, n_bins=10):
    bin_boundaries = np.linspace(0, 1, n_bins + 1)
    ece = 0.0
    
    for i in range(n_bins):
        mask = (y_prob >= bin_boundaries[i]) & (y_prob < bin_boundaries[i+1])
        if mask.sum() > 0:
            bin_accuracy = y_true[mask].mean()
            bin_confidence = y_prob[mask].mean()
            bin_weight = mask.sum() / len(y_true)
            ece += bin_weight * abs(bin_accuracy - bin_confidence)
    
    return ece

ece = expected_calibration_error(y_test, y_proba)
print(f"ECE: {ece:.4f}")

Complete Evaluation Example

from sklearn.metrics import (
    confusion_matrix, classification_report,
    precision_score, recall_score, f1_score,
    roc_auc_score, average_precision_score,
    brier_score_loss
)
import numpy as np

def evaluate_model(y_true, y_pred, y_proba=None):
    """Complete model evaluation."""
    
    # Confusion matrix
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    
    results = {
        'confusion_matrix': {'TN': tn, 'FP': fp, 'FN': fn, 'TP': tp},
        'accuracy': (tp + tn) / (tp + tn + fp + fn),
        'precision': precision_score(y_true, y_pred),
        'recall': recall_score(y_true, y_pred),
        'f1': f1_score(y_true, y_pred),
    }
    
    if y_proba is not None:
        results['roc_auc'] = roc_auc_score(y_true, y_proba)
        results['pr_auc'] = average_precision_score(y_true, y_proba)
        results['brier'] = brier_score_loss(y_true, y_proba)
    
    return results

# Example
results = evaluate_model(y_true, y_pred, y_proba)
for metric, value in results.items():
    if metric != 'confusion_matrix':
        print(f"{metric}: {value:.3f}")

Metric Selection Guide

Try It Yourself

• Data Leakage in ML — Avoid contamination in evaluation
• Train/Val/Test Sets — Split correctly before evaluating
• ML Production Failures — Why models fail after deployment
• Model Drift — Monitor for performance degradation

Further Reading

• Scikit-learn: Model Evaluation Metrics
• Scikit-learn: Precision-Recall Example
• Deepchecks: Understanding F1, ROC-AUC & PR-AUC
• BestWordz: Data Leakage in ML

Conclusion

Accuracy is a starting point, not the finish line. For any real-world classification problem, you need to understand precision, recall, F1, ROC-AUC, PR-AUC, and calibration.

The confusion matrix is your foundation. From it, all other metrics derive. Always start by examining the confusion matrix before looking at summary metrics.

The key principles:

• Precision — When false positives are costly
• Recall — When false negatives are costly
• F1 — When you need balance
• ROC-AUC — When you need threshold-independent evaluation
• PR-AUC — When data is imbalanced
• Calibration — When probabilities must be accurate

Choose the metric that matches your business problem, not the one that makes your model look best.

The best metric is the one that measures what actually matters.