Stacking (Model Stacking): Complete Guide — Principles, Examples and Python Implementation

Summary

Stacking (or model stacking, also called stacked generalization) is an advanced ensemble learning method that combines the predictions of several base models — called level-0 models — using a level-1 meta-model (or meta-learner). Unlike bagging which averages identical models or boosting which sequentially corrects errors, stacking learns how to best combine the predictions of heterogeneous models. This approach, introduced by David Wolpert in 1992, is one of the most powerful techniques in supervised machine learning, particularly prized in Kaggle competitions where it frequently allows gaining crucial positions in the leaderboard.

In this complete guide, we will explore the mathematical principles of stacking, its fundamental intuition, its practical implementation with scikit-learn (via StackingClassifier and StackingRegressor), its key hyperparameters, as well as four concrete use cases.

Mathematical Principle of Stacking

Stacking is based on a two-level architecture (sometimes more, hence the term “stacking”).

Level 0: Base Models

Consider a training set D = {(x₁, y₁), …, (xₙ, yₙ)} where xᵢ ∈ ℝᵖ are the features and yᵢ is the target. We train K different base models:

f₁, f₂, …, f_K

Each model f_k is trained on D and produces a prediction f_k(x) for a new observation x. These models are typically of different nature (algorithm, hyperparameters, feature subsets) to ensure prediction diversity — an essential condition for stacking success.

Level 1: The Meta-Model

The core idea is to build a meta-model g that takes as input the predictions of the K base models and produces the final prediction:

ŷ = g(f₁(x), f₂(x), …, f_K(x))

The meta-model g thus learns to intelligently weight the opinions of the different base models. For example, if Random Forest excels in certain regions of the feature space while SVM is better in others, g will learn to give more weight to the competent model in each region.

Cross-Validation to Avoid Data Leakage

A crucial problem arises here: if we train g on the same data used to train the f_k, the meta-model risks learning the systematic biases of the base models rather than their generalization ability. This is called data leakage.

The solution, introduced by Wolpert, relies on out-of-fold cross-validation:

1. We divide D into M folds D₁, D₂, …, D_M.
2. For each fold D_m and each model f_k:
   – We train f_k on D \ D_m (all data except fold m).
   – We predict on D_m only, obtaining out-of-fold predictions.
3. We assemble all out-of-fold predictions to form the meta-base Z, where each row i contains (f₁⁽⁻ⁱ⁾(xᵢ), f₂⁽⁻ⁱ⁾(xᵢ), …, f_K⁽⁻ⁱ⁾(xᵢ)).
4. We train the meta-model g on Z with the true labels y.

This procedure guarantees that the meta-model g has never seen the data on which the base models make their predictions during level-1 training. The result is an honest estimate of each model’s performance and an optimal combination without overfitting.

To produce the final prediction on new data, each base model f_k is retrained on the entirety of D, then their predictions are combined by g.

Intuition: The Committee of Experts and Their Project Manager

Imagine the following situation: a consulting firm needs to evaluate the value of a company. They convene three experts:

• The financial analyst (our Random Forest): excellent at spotting past trends and recurring patterns.
• The industry strategist (our Gradient Boosting): strong at understanding competitive dynamics and weak signals.
• The legal expert (our SVM): capable of detecting subtleties and edge cases that others miss.

Each gives their estimate. But instead of doing a simple average (that would be bagging), we bring in a project manager (the meta-model g). This project manager has experience: they know the financial analyst tends to overestimate tech companies, the strategist is excellent in mature sectors, and the legal expert is indispensable when there are ongoing disputes.

The project manager doesn’t just add up the opinions: they weight them contextually. Depending on the company’s profile, they place more or less trust in each expert. This is exactly what stacking does: the meta-model learns when to trust which model.

Python Implementation with scikit-learn

Complete Example: StackingClassifier

Here is a complete implementation using scikit-learn’s StackingClassifier, with three base models and a logistic regression as the meta-model:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_predict
from sklearn.ensemble import (
    RandomForestClassifier,
    GradientBoostingClassifier,
    StackingClassifier,
)
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report

# 1. Generate data
X, y = make_classification(
    n_samples=2000,
    n_features=20,
    n_informative=12,
    n_redundant=4,
    n_classes=2,
    random_state=42,
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# 2. Define base models (level 0)
estimators = [
    ("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
    ("gb", GradientBoostingClassifier(n_estimators=100, random_state=42)),
    ("svm", SVC(probability=True, random_state=42)),
]

# 3. Stacking with meta-model (level 1)
stacking_clf = StackingClassifier(
    estimators=estimators,
    final_estimator=LogisticRegression(max_iter=1000),
    cv=5,                     # 5-fold cross-validation
    stack_method="predict_proba",  # Uses probabilities
    n_jobs=-1,                # Parallelization
    passthrough=False,        # Does not add original features
)

# 4. Train and evaluate
stacking_clf.fit(X_train, y_train)
y_pred = stacking_clf.predict(X_test)

print("=== StackingClassifier ===")
print(f"Test accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(classification_report(y_test, y_pred))

# 5. Comparison with individual models
print("\n=== Individual performances ===")
for name, clf in estimators:
    clf.fit(X_train, y_train)
    pred = clf.predict(X_test)
    acc = accuracy_score(y_test, pred)
    print(f"{name:20s} : {acc:.4f}")

Demonstration of cross_val_predict

To understand the underlying cross-validation mechanism, here is how to manually reconstruct the meta-base with cross_val_predict:

from sklearn.model_selection import cross_val_predict

# Out-of-fold predictions for each base model
meta_features_train = np.column_stack([
    cross_val_predict(clf, X_train, y_train, cv=5, method="predict_proba")[:, 1]
    for _, clf in estimators
])

print(f"Shape of training meta-base: {meta_features_train.shape}")
# Result: (1600, 3) --- 1600 samples, 3 base models

# Train meta-model on these predictions
meta_model = LogisticRegression(max_iter=1000)
meta_model.fit(meta_features_train, y_train)

StackingRegressor

The same principle applies to regression via StackingRegressor:

from sklearn.ensemble import StackingRegressor
from sklearn.linear_model import Ridge, Lasso
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error

estimators_reg = [
    ("ridge", Ridge(alpha=1.0)),
    ("lasso", Lasso(alpha=0.1)),
    ("dt", DecisionTreeRegressor(max_depth=5)),
]

stacking_reg = StackingRegressor(
    estimators=estimators_reg,
    final_estimator=Ridge(alpha=1.0),
    cv=5,
)

stacking_reg.fit(X_train_reg, y_train_reg)
y_pred_reg = stacking_reg.predict(X_test_reg)
rmse = np.sqrt(mean_squared_error(y_test_reg, y_pred_reg))
print(f"Stacking RMSE: {rmse:.4f}")

Key Hyperparameters

estimators

List of (name, estimator) tuples defining the level-0 models. Diversity is key: use algorithms of fundamentally different nature. A stacking with three Random Forests will have little interest, as they will make similar errors.

estimators = [
    ("rf", RandomForestClassifier(n_estimators=200, max_depth=10)),
    ("gb", GradientBoostingClassifier(n_estimators=200, learning_rate=0.05)),
    ("svm", SVC(C=1.0, kernel="rbf", probability=True)),
    ("xgb", XGBClassifier(n_estimators=200, learning_rate=0.05)),  # With XGBoost
]

final_estimator

The meta-model that combines level-0 predictions. By default, scikit-learn uses LogisticRegression for classification and Ridge for regression. In practice:

• LogisticRegression / Ridge: default choice, robust, not prone to overfitting.
• GradientBoosting / XGBoost: more powerful but risk of overfitting if the dataset is small.
• Neural network: possible for very large datasets.

cv

Number of folds for cross-validation. Default is 5. Increasing to 10 reduces the variance of out-of-fold predictions at the cost of increased computation time. For small datasets (n < 1000), cv=10 is recommended.

stack_method

Determines which method of the base models is used as a feature for the meta-model:

• "predict_proba" (Classification): Class probabilities — recommended.
• "decision_function" (Classification): Raw score before probability transformation.
• "predict" (Classification/Regression): Direct prediction (hard classification).

For regression, stack_method is ignored since only predict exists.

passthrough

• passthrough=False (default): meta-model features are only the base model predictions.
• passthrough=True: the original features X are added to the base model predictions. This can be useful if the base models don’t capture all available information, but increases the risk of meta-model overfitting.

Advantages and Limitations of Stacking

Advantages

1. Superior performance: Stacking generally outperforms each individual model by leveraging their complementary strengths.
2. Algorithmic flexibility: Any estimators can be combined — trees, SVMs, neural networks, linear models.
3. Robustness: Less sensitive to overfitting than a single very complex model, thanks to integrated cross-validation.
4. No restrictive assumptions: Unlike boosting which assumes errors are sequentially correctable, stacking makes no assumptions about the error structure of base models.
5. Relative transparency: If the meta-model is linear (LogisticRegression), the coefficients reveal the relative importance of each base model.

Limitations

1. Computational complexity: Training K models in cross-validation multiplies computation time by K × cv (e.g., 3 models × 5 folds = 15 training runs at level 0).
2. Risk of meta-model overfitting on level-0 predictions, especially with a small dataset or an overly complex final_estimator.
3. Interpretation difficulty: The two-level architecture makes the global model hard to interpret, even if the meta-model is simple.
4. Sensitivity to base model quality: If all base models are mediocre, the meta-model cannot work miracles. Stacking amplifies performing diversity, it does not create performance.
5. Production maintenance: Deployment requires maintaining K + 1 models, increasing operational complexity.

4 Concrete Use Cases

1. Data Science Competitions (Kaggle)

Stacking is the secret weapon of Kaggle competitions. Winning teams frequently combine 5 to 10 heterogeneous models (Random Forest, XGBoost, LightGBM, CatBoost, neural networks) via a meta-model. On tabular datasets, two or three-level stacking can offer a 2 to 5% metric gain over the best individual model — enough to jump from 50th to 5th place.

2. Financial Fraud Detection

In fraud detection, individual models capture different signals: trees detect complex business rules, SVMs identify anomalies in subspaces, neural networks capture temporal patterns. Stacking combines these signals optimally, reducing both false positives (legitimate transactions blocked) and false negatives (undetected fraud).

3. Assisted Medical Diagnosis

Multiple algorithms analyze the same medical data (images, biomarkers, patient records): a CNN for imaging, a sequential model for vital sign time series, a tabular model for demographic and clinical data. Stacking aggregates these analyses into a single recommendation, more reliable than each isolated source — essential when a decision involves the patient’s life.

4. Demand Forecasting and Sales Prediction

In supply chain, different models capture different seasons and trends: Prophet for calendar seasonality, Random Forest for promotions and events, regression for underlying trends. Stacking produces a more robust consensus forecast, reducing both stockouts and overstocking.

Multi-Level Stacking

Nothing prevents further stacking: level-1 predictions can themselves serve as features for a level-2, and so on. This is called multi-level stacking or super learning. In practice:

• Level 1: 5 base models → meta-model A
• Level 2: Predictions of A + individual models → meta-model B (final)

This approach is common in competition but rarely justified in production, where the additional complexity brings only marginal gain.

Best Practices

1. Diversify base models: Choose fundamentally different algorithms. Three random forests with different seeds add little value.
2. Evaluate each model individually: Only include models that surpass a reasonable baseline. Remove “noisy” models.
3. Keep the meta-model simple: LogisticRegression or Ridge are excellent default choices. Avoid complex meta-models unless you have a lot of data.
4. Use nested cross-validation to honestly estimate final performance without optimistic bias.
5. Watch for class imbalance: Use stratify in the split and adapted metrics (F1-score, AUC-ROC, Average Precision) rather than simple accuracy.
6. Scale features for scale-sensitive models (SVM, regression) before training base models.

See Also

• How to Implement Binary Search in Python: A Step-by-Step Guide
• Energize Your Python Projects: Simulating Amoebas in a 3D Grid with Python