Explainable AI for Data Scientists: SHAP, LIME and Feature Importance

Your model predicts a customer will churn. The business asks: "Why?"

You shrug. The model is a black box.

Explainable AI (XAI) solves this problem. It opens the black box and shows which features drove each prediction.

This article compares 4 XAI methods with Python examples and helps you choose the right one.

The 4 Methods at a Glance

1. SHAP (SHapley Additive exPlanations)

SHAP is based on game theory. It calculates each feature's contribution to every prediction.

import shap
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Load data
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=1000, n_features=10, random_state=42)
feature_names = [f'feature_{i}' for i in range(10)]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# SHAP explanation
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Global feature importance
shap.summary_plot(shap_values[1], X_test, feature_names=feature_names)

# Single prediction explanation
shap.force_plot(
    explainer.expected_value[1],
    shap_values[1][0],
    X_test[0],
    feature_names=feature_names
)

SHAP Values Interpretation

# Get SHAP values for class 1 (positive class)
shap_vals = shap_values[1]

# For a single prediction
print("Feature contributions for first sample:")
for i, (name, val) in enumerate(zip(feature_names, shap_vals[0])):
    direction = "↑" if val > 0 else "↓"
    print(f"  {name}: {val:+.4f} {direction}")

# Positive SHAP value = pushes prediction toward positive class
# Negative SHAP value = pushes prediction toward negative class

SHAP Strengths

• Theoretically grounded — Based on Shapley values from game theory
• Consistent — Features always sum to the prediction difference
• Global + Local — Individual and aggregate explanations
• Visual — Beautiful summary and force plots

SHAP Limitations

• Slow — Computationally expensive for large datasets
• Memory intensive — Stores all feature combinations
• Assumes independence — May not handle correlated features well

2. LIME (Local Interpretable Model-agnostic Explanations)

LIME explains individual predictions by fitting a simple surrogate model around the prediction.

import lime
import lime.lime_tabular

# Create LIME explainer
explainer = lime.lime_tabular.LimeTabularExplainer(
    X_train,
    feature_names=feature_names,
    class_names=['Negative', 'Positive'],
    mode='classification'
)

# Explain a single prediction
exp = explainer.explain_instance(
    X_test[0],
    model.predict_proba,
    num_features=10
)

# Show explanation
print("LIME Explanation for first sample:")
for feature, weight in exp.as_list():
    print(f"  {feature}: {weight:+.4f}")

# Visual explanation
exp.show_in_notebook()

LIME Strengths

• Model-agnostic — Works with any model
• Intuitive — Easy to understand "why this prediction"
• Fast — Quick local explanations
• Flexible — Works with text, images, tabular

LIME Limitations

• Unstable — Different runs may give different explanations
• Local only — No global feature importance
• Hyperparameter sensitive — Kernel width affects results

3. Permutation Feature Importance

Permutation importance measures how much model performance decreases when a feature is randomly shuffled.

from sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt

# Calculate permutation importance
result = permutation_importance(
    model, X_test, y_test,
    n_repeats=10,
    random_state=42,
    scoring='f1'
)

# Sort by importance
sorted_idx = result.importances_mean.argsort()

# Plot
plt.figure(figsize=(10, 6))
plt.boxplot(
    result.importances[sorted_idx].T,
    vert=False,
    labels=[feature_names[i] for i in sorted_idx]
)
plt.title("Permutation Feature Importance")
plt.xlabel("F1 Score Decrease")
plt.tight_layout()
plt.show()

# Print results
print("Permutation Importance:")
for i in sorted_idx:
    print(f"  {feature_names[i]}: {result.importances_mean[i]:.4f} "
          f"± {result.importances_std[i]:.4f}")

Permutation Strengths

• Model-agnostic — Works with any model
• Intuitive — "How much does shuffling hurt?"
• Reliable — Less biased than tree importance
• Simple — Easy to implement and understand

Permutation Limitations

• Global only — No single-prediction explanations
• Correlated features — Importance spread across correlated features
• Computationally expensive — Multiple re-evaluations needed

4. Tree Feature Importance

Tree-based models have built-in feature importance (Gini or entropy-based).

import numpy as np

# Tree-based feature importance
importances = model.feature_importances_
std = np.std([tree.feature_importances_ for tree in model.estimators_], axis=0)

# Sort
sorted_idx = importances.argsort()

# Plot
plt.figure(figsize=(10, 6))
plt.barh(
    range(len(sorted_idx)),
    importances[sorted_idx],
    xerr=std[sorted_idx],
    align='center'
)
plt.yticks(range(len(sorted_idx)), [feature_names[i] for i in sorted_idx])
plt.title("Tree Feature Importance (Gini)")
plt.xlabel("Importance")
plt.tight_layout()
plt.show()

# Print results
print("Tree Feature Importance:")
for i in sorted_idx:
    print(f"  {feature_names[i]}: {importances[i]:.4f}")

Tree Strengths

• Very fast — Calculated during training
• No extra computation — Built into the model
• Works well — For tree-based models

Tree Limitations

• Trees only — Not applicable to other models
• Biased — Prefers high-cardinality features
• Impurity-based — Can be misleading
• No direction — Does not show positive/negative impact

When to Use Each Method

Complete Comparison

from sklearn.inspection import permutation_importance
import shap
import lime.lime_tabular

def compare_xai_methods(model, X_train, X_test, y_test, feature_names):
    """Compare all 4 XAI methods."""
    
    results = {}
    
    # 1. Tree Importance
    results['tree'] = dict(zip(
        feature_names,
        model.feature_importances_
    ))
    
    # 2. Permutation Importance
    perm_result = permutation_importance(
        model, X_test, y_test, n_repeats=10, random_state=42
    )
    results['permutation'] = dict(zip(
        feature_names,
        perm_result.importances_mean
    ))
    
    # 3. SHAP
    explainer = shap.TreeExplainer(model)
    shap_values = explainer.shap_values(X_test)
    results['shap'] = dict(zip(
        feature_names,
        np.abs(shap_values[1]).mean(axis=0)
    ))
    
    # 4. LIME (for single instance)
    lime_explainer = lime.lime_tabular.LimeTabularExplainer(
        X_train, feature_names=feature_names, mode='classification'
    )
    lime_exp = lime_explainer.explain_instance(
        X_test[0], model.predict_proba, num_features=len(feature_names)
    )
    results['lime'] = dict(lime_exp.as_list())
    
    return results

# Compare
results = compare_xai_methods(model, X_train, X_test, y_test, feature_names)

# Rank features by each method
for method, scores in results.items():
    print(f"\n{method.upper()} Rankings:")
    sorted_features = sorted(scores.items(), key=lambda x: abs(x[1]), reverse=True)
    for rank, (feat, score) in enumerate(sorted_features[:5], 1):
        print(f"  {rank}. {feat}: {score:.4f}")

Try It Yourself

• Model Evaluation Beyond Accuracy — Evaluate models properly before explaining
• Feature Engineering — Create features worth explaining
• ML Production Failures — Understand why models fail
• Model Drift — Monitor for performance degradation

Further Reading

• SHAP Documentation
• LIME Documentation
• Scikit-learn: Permutation Importance
• BestWordz: Model Evaluation Beyond Accuracy

Conclusion

Explainable AI is not optional. Regulations require it. Users demand it. Debugging needs it.

The 4 methods serve different purposes:

• SHAP — Theoretically grounded, comprehensive, but slow
• LIME — Fast local explanations, but unstable
• Permutation Importance — Simple, reliable, but global only
• Tree Importance — Very fast, but trees only and biased

The best practice: use multiple methods. If all methods agree on a feature's importance, you can be confident. If they disagree, investigate further.

If you cannot explain your model's predictions, you do not fully understand your model.