SHAP LIME — The Two Pillars of Explainable AI

Summary

In a world where machine learning models make increasingly complex decisions, understanding why an algorithm produced a given prediction has become as important as the prediction itself. Deep neural networks, random forests, and boosting models excel in accuracy, but they remain impenetrable black boxes. This is precisely where SHAP and LIME come in, two complementary approaches that make these models usable and transparent.

SHAP (SHapley Additive exPlanations) provides global and local explanations grounded in cooperative game theory, specifically Shapley values. LIME (Local Interpretable Model-agnostic Explanations) focuses on local interpretability by approximating the complex model with a simple linear model in the neighborhood of a given instance. Together, SHAP and LIME are the essential tools for any practitioner seeking to make their models interpretable and trustworthy.

Mathematical Principle

SHAP: Shapley Values Applied to Machine Learning

The contribution of each feature to an individual prediction is measured by the Shapley value, a concept from cooperative game theory developed by Lloyd Shapley in 1953, work that earned him the Nobel Prize in Economics.

The fundamental formula is:

$$\phi_i = \sum_{S \subseteq N \setminus {i}} \frac{|S|! \cdot (M – 1 – |S|)!}{M!} \cdot [f(S \cup {i}) – f(S)]$$

where:

• N is the set of all features of the model, of cardinality M
• S is any subset of features not containing feature i
• f(S) represents the prediction of the trained model on only the subset S of features
• [f(S ∪ {i}) − f(S)] is the marginal contribution of feature i when added to subset S
• |S|! · (M − 1 − |S|)! / M! is the weight coefficient assigned to this marginal contribution, ensuring all possible combinations are considered with fair weighting

This formula has four fundamental mathematical properties that make it unique:

1. Efficiency (or additivity): the sum of Shapley values for all features exactly equals the difference between the real prediction and the average prediction: Σ φᵢ = f(x) − E[f(x)]. This is an absolute guarantee that the entire prediction is fully explained.
2. Symmetry: if two features contribute identically to all possible coalitions, they receive exactly the same Shapley values. This property ensures fundamental fairness of treatment.
3. Linearity: if two models are combined, the Shapley value of the combination is the sum of the individual Shapley values. This property enables analysis of composite models.
4. Dummy (nullity): a feature that brings no marginal contribution to any coalition receives a strictly zero Shapley value. Useless variables are automatically identified.

The computational challenge is however considerable: computing exact Shapley values requires evaluating 2^M possible combinations, which quickly becomes impossible for models with hundreds of features. This is why SHAP uses efficient approximation algorithms such as KernelSHAP (a weighted linear regression), TreeSHAP (specially optimized for tree-based models like XGBoost and LightGBM), and DeepSHAP (adapted to deep neural networks via backpropagation of contributions).

LIME: Local Approximation by Interpretable Model

LIME is based on an idea that is remarkably simple yet profoundly powerful: rather than trying to explain the entire complex model, we focus only on the immediate neighborhood of the instance we want to understand.

Formally, LIME seeks to find an explanation function g that minimizes the following objective:

$$\xi(x) = \underset{g \in G}{\text{argmin}} \quad L(f, g, \pi_x) + \Omega(g)$$

where:

• f is the complex (black-box) model we want to explain, whether it is a neural network, a random forest, or any other model
• g is the interpretable model, belonging to the class G of interpretable functions (usually a linear model with a small number of non-zero coefficients, LASSO-type or simple regression)
• L(f, g, π_x) is the local fidelity measure, i.e., how well the simple model g imitates the behavior of the complex model f in the neighborhood of instance x. The lower this value, the better the local approximation.
• π_x is a proximity weighting function that assigns higher weights to perturbed samples close to the original instance x than to those far from it. Typically, an exponential kernel is used: π_x(z) = exp(−d(x,z)² / σ²), where d is a distance (often cosine or Euclidean distance) and σ is a bandwidth parameter.
• Ω(g) is a complexity penalty that ensures the explanation model g remains simple and interpretable. For example, Ω can penalize the number of non-zero features in g (L1-type sparsity) or the magnitude of coefficients.

The algorithmic process of LIME proceeds in four successive steps:

1. Perturbation sampling: a large number of perturbed data points are generated by randomly modifying the features of the instance x to be explained. For tabular data, values are replaced with samples from the marginal distribution. For text, certain words are randomly removed. For images, superpixels are masked.
2. Obtaining predictions: for each perturbed sample, the prediction of model f on that modified sample is obtained. These predictions constitute the targets that the interpretable model g will seek to reproduce.
3. Distance weighting: each perturbed sample is assigned a weight proportional to its similarity with the original instance x. Samples very close to x have a high weight, meaning g must predict them particularly well.
4. Fitting the interpretable model: a weighted linear model (usually a linear regression with LASSO regularization) is trained on the perturbed samples, producing directly interpretable coefficients that constitute the final local explanation.

Intuition Everyone Can Understand

SHAP: Fairly Splitting a Tip

Imagine the following situation: you go out to dinner with four friends and you collectively decide to leave a tip of fifty euros. The natural question is immediate — how to split this tip fairly among the five waiters who took turns during your evening?

The naive method would be to divide equally — ten euros per person. But this approach is deeply unfair: Marie served the appetizer and main course, Thomas only brought the drinks twenty minutes late, and Sophie handled the dessert and the bill while the others were absent.

Shapley values solve this problem with irreproachable mathematical rigor. Each waiter’s contribution is evaluated by considering all possible combinations of waiters:

• What is the difference in customer satisfaction when Marie is present versus when she is absent, for each possible team? This is measured systematically.
• All these marginal contributions are averaged, weighting each situation by its statistical importance.
• The result is a perfectly fair distribution: each waiter receives exactly what their unique contribution brings to the overall quality of service.

Transposed to machine learning, the “tip” represents the difference between the real prediction and the average prediction. Each feature (each “waiter”) receives its fair share of responsibility for this difference. If a variable increases the prediction significantly and consistently across all possible coalitions, it gets a high SHAP value. If it changes nothing, its value is zero.

LIME: The Mechanic Who Isolates Each Part

Now, imagine this classic scene: your car is making a strange and worrying noise that you cannot identify. You take the vehicle to an experienced mechanic. What does he actually do?

1. He starts the engine and listens carefully to the abnormal noise — this is the original model prediction being observed.
2. He tests each component individually: he removes the belt, runs the engine again, notes whether the noise persists. He puts the belt back, tests the alternator, and repeats the process systematically.
3. He records his observations for each individual test: “with the belt removed, the noise decreases by 70%” — this means the belt contributes significantly to the problem.
4. He formulates a simple and understandable diagnosis: “it’s mainly the accessory belt and the tensioner causing this noise, in a 70/30 proportion.” This explanation is linear and transparent — you can understand it immediately without being a mechanic.

This is exactly the principle of LIME: we perturb the model’s input (we “remove parts”), observe the prediction changes (we “listen to the resulting noise”), and build a simple local explanation (the “mechanic’s diagnosis”) that is faithful to the model’s behavior in that specific neighborhood.

Complete Python Implementation

SHAP — Concrete Examples with KernelExplainer

Let’s start with the practical implementation of SHAP with a standard scikit-learn classification model:

import shap
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Load data — breast tumor classification
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)

# Train a random forest — a precise black-box type model
model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(X_train, y_train)

# Model accuracy to verify quality before explanation
accuracy = model.score(X_test, y_test)
print(f"Model accuracy: {accuracy:.4f}")

# Initialize the KernelExplainer — an agnostic method
# that works with any predictive model
# A background sample is used to estimate the reference value
explainer = shap.KernelExplainer(
    model.predict_proba,
    shap.sample(X_train, 100),  # Representative sample to speed up
    link="logit"
)

# Compute SHAP values for the first 5 test instances
# KernelSHAP is precise but computationally expensive
shap_values = explainer.shap_values(X_test[:5])

# --- Visualization 1: Global summary of contributions ---
# The summary plot ranks features by average importance
# and shows the direction (positive/negative) of their influence
shap.summary_plot(shap_values, X_test[:5], feature_names=data.feature_names)
plt.title("SHAP Summary — Global Feature Impact")
plt.tight_layout()
plt.savefig("shap_summary_global.png", dpi=150, bbox_inches="tight")
plt.close()

# --- Visualization 2: Force plot for an individual instance ---
# The force plot visually shows how each feature
# pushes the prediction towards class 0 or class 1
instance_idx = 0
shap.force_plot(
    explainer.expected_value[1],
    shap_values[1][instance_idx],
    X_test[instance_idx],
    feature_names=data.feature_names,
    matplotlib=True
)
plt.title(f"SHAP Force Plot — Instance {instance_idx}")
plt.tight_layout()
plt.savefig(f"shap_force_instance_{instance_idx}.png", dpi=150, bbox_inches="tight")
plt.close()

# --- Visualization 3: Dependence plot ---
# Shows the effect of a single feature on the prediction
shap.dependence_plot(
    "mean radius",
    shap_values[1],
    X_test,
    interaction_index="mean concave points",
    feature_names=data.feature_names
)
plt.title("SHAP Dependence — Mean Radius")
plt.savefig("shap_dependence_radius.png", dpi=150, bbox_inches="tight")
plt.close()

For tree-based models (XGBoost, LightGBM, random forests), SHAP offers a much faster algorithm called TreeSHAP which is exact (no approximation) and hundreds of times faster than KernelSHAP:

# TreeSHAP — specifically optimized for decision trees
tree_explainer = shap.TreeExplainer(model)
tree_shap_values = tree_explainer.shap_values(X_test)

# The summary plot with TreeSHAP works identically
# but the computation is exponentially faster
shap.summary_plot(tree_shap_values, X_test, feature_names=data.feature_names, show=False)
plt.title("SHAP Summary via TreeExplainer")
plt.savefig("shap_tree_summary.png", dpi=150, bbox_inches="tight")
plt.close()

LIME — Local Explanation with LimeTabularExplainer

The implementation of LIME is remarkably elegant and intuitive:

import lime
import lime.lime_tabular
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Data preparation identical to the SHAP example
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)

model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(X_train, y_train)

# Initialize the LIME explainer for tabular data
# Feature and class names must be specified
# as well as the mode (classification or regression)
lime_explainer = lime.lime_tabular.LimeTabularExplainer(
    training_data=X_train,
    feature_names=data.feature_names,
    class_names=data.target_names.astype(str).tolist(),
    mode='classification',
    verbose=True,  # Shows explanation process details
    random_state=42
)

# --- Explain an individual instance ---
instance_idx = 0
explanation = lime_explainer.explain_instance(
    X_test[instance_idx],
    model.predict_proba,
    num_features=10,  # Maximum number of features in the explanation
    top_labels=1,      # Explain only the predicted class
)

# Display the most important results
print(f"Actual prediction: class {y_test[instance_idx]}")
print(f"Probabilities: {model.predict_proba([X_test[instance_idx]])}")
print(f"Local fidelity score (R²): {explanation.score:.4f}")
print("\nContributions of the most important features:")
for feature, contribution in explanation.as_list():
    print(f"  {feature}: {contribution:+.4f}")

# Save the visualization as interactive HTML
explanation.save_to_file(f"lime_explanation_instance_{instance_idx}.html")

# Display in a Jupyter notebook
# explanation.show_in_notebook()

# --- Example with text data ---
# LIME also works with text, images, and sequential data
# from lime.lime_text import LimeTextExplainer
# text_explainer = LimeTextExplainer(class_names=categories)
# text_explanation = text_explainer.explain_instance(
# text_document, classifier.predict_proba, num_features=15
# )

Direct Comparison of Both Approaches

# SHAP vs LIME comparison script
import pandas as pd

shap_df = pd.DataFrame(
    np.abs(tree_shap_values).mean(axis=0),
    columns=['mean_absolute_shap_value']
)

lime_df = pd.DataFrame(
    explanation.as_map(),
    columns=['feature', 'lime_contribution']
)

# Merge and compare feature rankings
# for a given instance
comparison = pd.DataFrame({
    'shap_absolute': np.abs(tree_shap_values[instance_idx]),
    'lime_contribution': 
}, index=data.feature_names)

print("Comparison of relative importances:")
print(comparison.sort_values('shap_absolute', ascending=False).head(10))

Computational Performance

One of the decisive criteria in choosing between SHAP and LIME is computation time:

Times measured on a modern processor with 30 features and a forest of 200 trees.

Advantages and Limitations

SHAP — Advantages

Theoretically sound and rigorous: Shapley values are the only attribution method that simultaneously satisfies the four fundamental mathematical properties (efficiency, symmetry, linearity, and dummy). No other method — permutation importance, information gains, or raw coefficients — guarantees this absolute mathematical consistency.

Global consistency: SHAP values are comparable across different instances and different models. If a feature has a SHAP value of +0.5 in one model and −0.2 in another, we can conclude with certainty that the first occurrence strongly pushes toward a positive prediction while the second opposes that prediction. This comparability is unique to SHAP.

Rich and varied visualizations: The SHAP library offers a complete ecosystem of visualizations — the summary plot for a global overview, the force plot for individual instances, the dependence plot for analyzing interactions, the decision plot for tracking the complete prediction path.

Optimized for tree-based models: TreeSHAP is exact and extremely fast, making explanation of random forests and boosting models nearly as fast as the prediction itself.

SHAP — Limitations

Prohibitive computational cost for KernelSHAP: for non-tree models (deep neural networks, SVMs, nonlinear regressions), KernelSHAP requires a considerable number of model evaluations, which quickly becomes impractical beyond 50-100 features.

Incorrect causal interpretation: SHAP measures statistical associations, not causal relationships. A feature can get a high SHAP value because it is correlated with another truly determining feature, not because it itself causes the outcome. This confusion is frequent and dangerous.

Dependence on the background distribution: results depend on the choice of the background sample used to estimate the reference value. An unrepresentative sample can significantly bias all computed SHAP values.

LIME — Advantages

Model-agnostic: LIME works with absolutely any predictive model — neural networks, SVMs, random forests, ensemble models, proprietary models accessible only via API. No knowledge of the internal architecture is necessary, which is a considerable advantage.

Extremely intuitive: explanations in the form of weighted linear coefficients are immediately understandable by non-experts. “Income increases the probability of approval by 0.3” is a sentence that a credit officer can understand and use without prior mathematical training.

Remarkable flexibility: LIME naturally adapts to different types of data — text (word removal), images (superpixel masking), time series, tabular data — with appropriate perturbations for each modality.

Local speed: to explain a single instance, LIME is significantly faster than KernelSHAP because it does not require exhaustive evaluation of all possible coalitions.

LIME — Limitations

Explanation instability: due to the random sampling of perturbations, two runs of LIME on the same instance can produce slightly different results. This variability can undermine end-user confidence in the explanations produced.

Absence of global guarantees: LIME only explains the immediate neighborhood of an instance. A LIME explanation can differ radically between two neighboring instances if the model exhibits pronounced local nonlinear behavior. LIME gives no overall view of the model.

Parameter sensitivity: the choice of the number of perturbed samples, the bandwidth of the weighting kernel (σ), and the maximum complexity of the interpretable model considerably influences the results obtained. No default parameter universally suits all situations.

Four Detailed Concrete Use Cases

1. Bank Credit Audit — European Regulatory Compliance

In the strict context of the General Data Protection Regulation (GDPR) and Article 22 relating to automated decisions, European banks must be able to explain each credit denial to their clients. A credit scoring model based on XGBoost uses approximately 150 features — income, payment history, existing debts, age, profession, and many others — to calculate a risk score.

When a client is denied a mortgage, the bank must communicate the main reasons for this denial. SHAP provides exactly this: for each denied application, a precise report is generated indicating that the score was decreased by −120 points due to an unfavorable payment history, by −45 points due to a debt ratio deemed too high, but increased by +30 points thanks to a stable salary and +15 points thanks to significant professional seniority.

This explanation is legally compliant because it is based on a mathematically grounded method (Shapley values), reproducible (two identical calculations give exactly the same result), and understandable by the end client who receives a concrete and actionable list of reasons.

2. AI-Assisted Medical Diagnosis

AI models used in radiology to detect anomalies — pulmonary nodules, breast lesions, complex fractures on X-rays — must imperatively provide transparent justifications so that radiologists can validate or invalidate predictions with complete understanding.

Grad-CAM (a SHAP variant adapted to deep convolutional neural networks) produces saliency maps that highlight the exact regions of the image that the neural network considers most determinative for its prediction. The radiologist then sees a visual overlay where relevant areas appear in intense red on the original X-ray.

For tabular data from electronic medical records — blood test results, family history, prescribed medications — SHAP identifies the individual risk factors that most contributed to the diagnosis. For example: “This patient has a high risk of diabetes primarily because of their glycated hemoglobin level (+0.45 SHAP), their body mass index (+0.30 SHAP), and their family history (+0.20 SHAP).”

3. Financial Transaction Fraud Detection

Fraud detection systems analyze millions of transactions in real time by examining hundreds of signals — amount, geographic location, transaction time, customer history, deviation from usual behavior, execution speed, and other indicators.

LIME is particularly suited to this scenario because each blocked fraudulent transaction requires rapid and targeted investigation. The explanation must be instantaneous and actionable for analysts reviewing alerts: “This transaction of 2,500 euros made at 3 a.m. from a terminal located in São Paulo while the customer is usually in Paris was flagged as fraudulent primarily due to: abnormal geographic distance (LIME contribution: +0.72), unusual transaction time (LIME contribution: +0.35), and amount significantly higher than the customer’s usual average (LIME contribution: +0.28).”

This explanation allows the analyst to immediately understand why the transaction was flagged and to make an informed decision in a few seconds, thereby reducing both costly false positives and devastating missed fraud.

4. Predictive Maintenance in Manufacturing

In modern factories, IoT (Internet of Things) sensors continuously collect data on hundreds of machine parameters — bearing temperature, vibrations, hydraulic pressure, electrical power consumption, sound vibration quality — to predict potential failures before they occur.

SHAP identifies the most determinative precursor signals for each type of failure. For example, to predict a bearing failure on a turbine, SHAP analysis reveals that high-frequency axial vibration is the dominant predictor (mean SHAP value of +0.58), followed by the evolution of lubricant temperature over the last 48 hours (+0.41) and abnormal power consumption (+0.25).

This analysis not only makes it possible to predict the failure, but above all to understand its root cause and plan targeted maintenance interventions, thereby reducing unplanned production downtime by up to 40% and saving millions of euros in downtime and emergency repair costs.

See Also

• Solving the Maximum Subarray Interview Problem with Python
• Implementing Dekker’s Algorithm in Python: Efficient Thread Synchronization