Bagging (Bootstrap Aggregating): Complete Guide — Principles, Examples, and Python Implementation

Summary

Bagging (Bootstrap Aggregating) is an ensemble method introduced by Leo Breiman in 1996. Its principle is simple yet powerful: train multiple identical models on bootstrap samples (draws with replacement) from the training set, then aggregate their predictions by mean (regression) or majority vote (classification). Bagging significantly reduces the variance of unstable models — such as decision trees — without increasing bias. It forms the theoretical foundation of many modern ensemble methods, including Random Forest.

Mathematical principle of bootstrap bagging

Bootstrap sampling

Let a training set D = {(x₁, y₁), …, (xₙ, yₙ)} of size n. Bootstrap sampling consists of generating B samples D₁, D₂, …, D_B of the same size n, each obtained by drawing with replacement from D.

For each sample D_b, a model f_b is trained. After B training rounds, we have an ensemble of models {f₁, f₂, …, f_B}.

Prediction aggregation

In regression, the final prediction is the arithmetic mean:

ŷbagging(x) = (1/B) × Σ{b=1}^{B} f_b(x)

In classification, majority vote is used: the predicted class is the one that receives the most votes among the B models.

Variance reduction

Suppose each model f_b(x) has the same variance σ² and that the models are independent (approximation). The variance of the aggregated predictor is:

Var(ŷ_bagging) = σ² / B

Variance therefore decreases proportionally to the number B of models. In practice, the models are not perfectly independent (they share the same original dataset), but the variance reduction remains substantial — especially for high-variance models such as unpruned decision trees.

Important note: bagging has little effect on low-variance, high-bias models (linear regression, for example). Its effectiveness is maximal for unstable algorithms: those whose predictions vary strongly in response to small changes in the training data.

Bootstrap inclusion probability

Each observation has a probability of 1 − (1 − 1/n)ⁿ ≈ 1 − e⁻¹ ≈ 0.632 of being selected in a given bootstrap sample. Approximately 63.2% of the original data appears in each sample; the remaining 36.8% constitutes the Out-of-Bag (OOB) sample, usable for error estimation without a separate validation set.

Intuition: why bootstrap bagging works

Imagine consulting B independent doctors for a complex diagnosis. Each doctor has their own reasoning, their own biases, their own blind spots. Individually, each can be wrong. But if you take the majority decision among the B opinions, the collective error is generally less than that of a single doctor.

Bootstrap bagging applies exactly this logic to machine learning models:

• Each bootstrap sample is each model’s unique “lived experience.”
• The models learn slightly different patterns because they see different subsets of the data.
• By aggregating, we smooth the idiosyncratic errors of each model.

It is the algorithmic equivalent of the wisdom of crowds: the average of many imperfect opinions is often better than the most enlightened opinion taken in isolation.

Python implementation with scikit-learn

BaggingClassifier on DecisionTreeClassifier

Let’s compare a single decision tree with a bagged ensemble of 50 trees:

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report
import numpy as np

# Synthetic data
X, y = make_classification(
    n_samples=5000, n_features=20, n_informative=15,
    n_redundant=5, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Single decision tree (unlimited depth)
tree = DecisionTreeClassifier(random_state=42)
tree.fit(X_train, y_train)
tree_pred = tree.predict(X_test)
tree_acc = accuracy_score(y_test, tree_pred)
print(f"Single tree        : {tree_acc:.4f}")

# Bagging of 50 decision trees
bagging = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=50,
    max_samples=1.0,
    bootstrap=True,
    random_state=42,
    n_jobs=-1
)
bagging.fit(X_train, y_train)
bag_pred = bagging.predict(X_test)
bag_acc = accuracy_score(y_test, bag_pred)
print(f"Bagging (50 trees) : {bag_acc:.4f}")

# Comparison with Random Forest
rf = RandomForestClassifier(n_estimators=50, random_state=42, n_jobs=-1)
rf.fit(X_train, y_train)
rf_acc = accuracy_score(y_test, rf.predict(X_test))
print(f"Random Forest      : {rf_acc:.4f}")

The difference is often striking: a deep tree overfits and may achieve decent accuracy on training but poor accuracy on test. Bootstrap bagging reduces this overfitting by averaging the predictions of 50 trees that each learned on slightly different data.

BaggingRegressor for regression

from sklearn.datasets import make_regression
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor
from sklearn.metrics import mean_squared_error, r2_score

# Regression data
X_reg, y_reg = make_regression(
    n_samples=3000, n_features=15, noise=15.0, random_state=42
)
X_train_r, X_test_r, y_train_r, y_test_r = train_test_split(
    X_reg, y_reg, test_size=0.2, random_state=42
)

# Single tree
tree_reg = DecisionTreeRegressor(random_state=42)
tree_reg.fit(X_train_r, y_train_r)
print(f"Single tree (MSE)  : {mean_squared_error(y_test_r, tree_reg.predict(X_test_r)):.2f}")

# BaggingRegressor
bag_reg = BaggingRegressor(
    estimator=DecisionTreeRegressor(),
    n_estimators=100,
    random_state=42
)
bag_reg.fit(X_train_r, y_train_r)
bag_pred_r = bag_reg.predict(X_test_r)
print(f"Bagging (MSE)      : {mean_squared_error(y_test_r, bag_pred_r):.2f}")
print(f"Bagging (R²)       : {r2_score(y_test_r, bag_pred_r):.4f}")

Out-of-Bag (OOB) estimation

The Out-of-Bag sample allows estimating the generalization error without a separate validation set. Each observation is predicted by the models that did not see it during training:

bagging_oob = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=100,
    oob_score=True,
    random_state=42,
    n_jobs=-1
)
bagging_oob.fit(X_train, y_train)

print(f"OOB score  : {bagging_oob.oob_score_:.4f}")
print(f"Test score : {bagging_oob.score(X_test, y_test):.4f}")

The OOB score is an unbiased estimate of generalization performance, comparable to cross-validation but much faster since it reuses existing training.

Visualizing bagging convergence

import matplotlib.pyplot as plt

# Convergence analysis as a function of the number of estimators
scores = []
for n in range(1, 101):
    model = BaggingClassifier(
        estimator=DecisionTreeClassifier(),
        n_estimators=n,
        random_state=42
    )
    model.fit(X_train, y_train)
    scores.append(accuracy_score(y_test, model.predict(X_test)))

plt.figure(figsize=(10, 6))
plt.plot(range(1, 101), scores, marker='o', markersize=3,
         linewidth=1.5, color='steelblue')
plt.axhline(y=tree_acc, color='red', linestyle='--',
            label=f'Single tree ({tree_acc:.3f})')
plt.xlabel("Number of estimators (B)")
plt.ylabel('Accuracy')
plt.title('Bootstrap bagging convergence as a function of B')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

This curve typically shows rapid improvement up to B ≈ 20-30, then a plateau where adding more estimators no longer provides significant gains.

Key hyperparameters of bootstrap bagging

Practical tuning guide

• Start with n_estimators=50 to 100. Increasing beyond 100 rarely provides significant gains.
• max_samples=0.8 can improve model diversity at the cost of a slight decrease in individual performance.
• max_features < 1.0 progressively transforms bootstrap bagging into Random Forest: it is the double randomization mechanism (samples × features) that makes Random Forest so powerful.
• Enable oob_score=True to get a free estimate of generalization during training.

Advantages of bootstrap bagging

1. Massive variance reduction: especially effective for unstable models (deep decision trees, k-NN with small k, neural networks).
2. Trivial parallelization: each f_b model is trained independently of the others — bootstrap bagging adapts perfectly to distributed computing with n_jobs=-1.
3. No increased overfitting: unlike boosting, bagging does not overfit when B is increased.
4. Free OOB estimation: the Out-of-Bag sample provides integrated validation at no additional cost.
5. Conceptual simplicity: the principle is easy to explain and implement.

Limitations of bootstrap bagging

1. Ineffective for stable models: linear regression, logistic regression, or linear SVMs benefit little from bootstrap bagging because their variance is already low.
2. Reduced interpretability: an ensemble of 100 trees is far less interpretable than a single tree.
3. Computational cost: training B models costs B times more in time and memory.
4. No bias correction: if base models have high bias (underfitting), bagging does not correct it — it only reduces variance.
5. Less powerful than boosting: on many benchmarks, gradient boosting and XGBoost surpass simple bagging in raw accuracy, at the cost of greater complexity.
6. No feature subsampling by default: unlike Random Forest, standard bagging uses all features for each model, which limits ensemble diversity.

4 real-world use cases of bootstrap bagging

1. Banking fraud detection

Fraudulent transactions are rare and data is imbalanced. A single decision tree tends to overfit on patterns specific to the training set. Bagging 100 trees, each trained on a different bootstrap subsample, captures a wider variety of fraud patterns while reducing false positives:

bagging_fraud = BaggingClassifier(
    estimator=DecisionTreeClassifier(max_depth=8),
    n_estimators=100,
    max_samples=0.7,
    oob_score=True,
    random_state=42
)
bagging_fraud.fit(X_train_fraud, y_train_fraud)

2. Real estate price prediction

Real estate data contains many non-linear interactions between variables (neighborhood × area × year of construction). A BaggingRegressor with 200 trees captures these interactions better than a linear model, while remaining more robust than a single tree prone to overfitting.

3. Medical diagnosis assistance

The multiple-doctor analogy is literal here: training 50 classifiers on bootstrap samples of medical data allows combining different “perspectives” on the same symptoms, reducing the risk that an anomalous pattern in the training set influences the final diagnosis. The OOB score provides rigorous validation without reducing the size of the training set.

4. Text classification with weak classifiers

Bootstrap bagging can be applied to any classifier. For example, individually mediocre Naïve Bayes classifiers on a text corpus can achieve remarkable accuracy when 50 are aggregated by majority vote — each classifier capturing different aspects of vocabulary through bootstrap samples.

Bootstrap bagging in the ensemble method ecosystem

Bagging occupies a central place in the ensemble method family:

• Random Forest = bagging + random feature selection (article #025). This is the most famous and most widely used case of bootstrap bagging.
• AdaBoost = adaptive weighting of misclassified samples (article #030). Unlike bagging, samples are drawn without replacement with adjusted weights.
• Gradient Boosting = sequential learning of residual errors (article #027). Each model corrects the errors of the previous one, whereas bagging trains all models in parallel.

The fundamental distinction is clear: bagging seeks to reduce variance by aggregating independent models, while boosting seeks to reduce bias by sequentially learning from errors.

Conclusion

Bootstrap bagging is one of the most elegant ideas in modern machine learning. By leveraging the power of averaging over models trained on different bootstrap samples, it transforms unstable algorithms into robust and reliable predictors. While more sophisticated methods like gradient boosting surpass it in raw accuracy on certain benchmarks, bagging remains indispensable for its simplicity, native parallelization, and resistance to overfitting.

For any problem where a single decision tree overfits, bootstrap bagging is the first improvement to try — often with remarkable results.

See also

• Solving the Maximum Subarray Interview Problem with Python
• Implementing Dekker’s Algorithm in Python: Efficient Thread Synchronization