Bagging: Complete Guide — Bootstrap Aggregating

Summary

Bagging (short for Bootstrap Aggregating) is one of the most fundamental and effective ensemble techniques in machine learning. Introduced by Leo Breiman in 1996, Bagging reduces the variance of a predictive model by training multiple versions of the same algorithm on bootstrap subsamples drawn from the training dataset, then aggregating their predictions by majority vote (classification) or by averaging (regression). This method is particularly powerful when applied to high-variance models, such as deep decision trees. Understanding Bagging is essential for anyone looking to master modern ensemble methods.

Mathematical Principle of Bagging

Bootstrap sampling

The mathematical foundation of Bagging relies on the bootstrap technique, introduced by Bradley Efron in 1979. The principle is as follows: from a training dataset containing N observations, B bootstrap samples are generated. Each bootstrap sample is obtained by sampling with replacement of size N from the N original observations.

The key mathematical property of this sampling is: the probability that a given observation is excluded from a particular bootstrap sample is:

P(observation excluded) = (1 – 1/N)^N

When N is sufficiently large, this probability converges to a remarkable constant:

(1 – 1/N)^N → 1/e ≈ 0.3679 ≈ 37%

This means that approximately 37% of the original observations are never present in a given bootstrap sample. These observations, absent from a particular model’s training, constitute what is called the out-of-bag (OOB) data. They play a crucial role in internal model evaluation, without requiring a separate validation set.

Conversely, approximately 63% of observations are present in each bootstrap sample, with some appearing multiple times. This natural redundancy gives each model a slightly different perspective on the data.

Aggregating predictions

Once the B models are trained on their respective bootstrap samples, Bagging aggregates their predictions as follows:

In classification: majority vote

For a new observation x, each model b produces a class prediction C_b(x). The final prediction is the class that receives the most votes:

C_final(x) = argmax_c Σ_b I(C_b(x) = c)

where I(·) is the indicator function that equals 1 if the condition is true, 0 otherwise. Each model contributes equally to the vote, with no particular weighting.

In regression: arithmetic mean

For a new observation x, each model b produces a numerical prediction f_b(x). The final prediction is the average of all predictions:

f_final(x) = (1/B) Σ_b f_b(x)

This simple average is both elegant and effective: it smooths out the individual errors of each model.

Variance reduction: theoretical analysis

The fundamental benefit of Bagging is variance reduction. To understand this mechanism, consider a set of k models whose predictions each have a variance σ², and whose pairwise correlation is ρ (rho).

The variance of the average of these k correlated models is expressed as:

Var(average_of_k_models) = ρ·σ² + (1 – ρ)·σ²/k

This formula reveals several essential insights:

1. When k → ∞ (many models): the second term tends toward zero, and the variance converges to ρ·σ². The residual variance is therefore proportional to the correlation between models. If the models are perfectly correlated (ρ = 1), Bagging provides no benefit. This is why model diversity is crucial.
2. When ρ is low (highly diversified models): variance is considerably reduced, by a factor that can reach k. This is the ideal situation that the bootstrap seeks to create by presenting each model with a different subset of the data.
3. The trade-off: ideally, you want individually performant models that are also as uncorrelated as possible. The bootstrap achieves this trade-off naturally: each model sees 63% of the data, but not the same 63%.

In summary, Bagging is all the more effective when the base models are unstable (i.e., sensitive to small variations in the training set). Deep decision trees are precisely in this category: a slight modification of the training data can radically change the tree structure. This is why Bagging excels with decision trees, and it is also the founding principle of Random Forests.

Intuition: the wisdom of crowds

To understand Bagging intuitively, imagine the following situation: instead of asking the same complex question to one very intelligent but sometimes erratic person, you ask that same question to one hundred slightly different people and take the majority answer.

This analogy captures the essence of Bagging:

• One intelligent but erratic person = a powerful but high-variance model (a deep decision tree, for example). It can give excellent answers, but it is also likely to make spectacular mistakes depending on the data it saw during training.
• One hundred slightly different people = one hundred models of the same type, but each trained on a slightly different subsample of the data. Each one makes errors, but not the same errors.
• The majority answer = aggregation by vote or by average. When errors are independent or weakly correlated, they cancel each other out, and the collective answer is more reliable than any individual answer.

This phenomenon is well known in science as the wisdom of crowds. In 1906, Francis Galton demonstrated at a county fair that the average of 787 villagers’ weight estimates of an ox was within 1% of the actual weight. No individual had been as precise as the crowd as a whole.

Bagging applies exactly this principle to predictive models: the diversity of perspectives compensates for individual weaknesses, and aggregation produces a prediction that is more robust and more reliable than that of any single model taken in isolation.

Python Implementation with scikit-learn

Here is a complete implementation of Bagging with scikit-learn, using a DecisionTreeClassifier (a decision tree stump, i.e., a tree of depth 1) as the base model. We will compare the performance of Bagging with that of a single decision tree.

# ===========================================================================
# 130-bagging-implementation.py
# Illustration of Bagging with scikit-learn
# ===========================================================================

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, classification_report
from sklearn.datasets import make_classification
import matplotlib.pyplot as plt

# ---------------------------------------------------------------------------
# 1. Generating a synthetic dataset
# ---------------------------------------------------------------------------
X, y = make_classification(
    n_samples=2000,
    n_features=20,
    n_informative=10,
    n_redundant=5,
    n_classes=2,
    flip_y=0.1,          # 10% label noise
    class_sep=0.7,
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

print(f"Training set: {X_train.shape[0]} examples")
print(f"Test set       : {X_test.shape[0]} examples")
print(f"Class distribution: {np.bincount(y_train)}")
print()

# ---------------------------------------------------------------------------
# 2. Single decision tree (reference)
# ---------------------------------------------------------------------------
single_tree = DecisionTreeClassifier(
    max_depth=None,       # full tree (unpruned)
    random_state=42
)
single_tree.fit(X_train, y_train)

train_acc_tree = accuracy_score(y_train, single_tree.predict(X_train))
test_acc_tree = accuracy_score(y_test, single_tree.predict(X_test))

print("=" * 60)
print("SINGLE DECISION TREE (reference)")
print(f"  Training accuracy : {train_acc_tree:.4f}")
print(f"  Test accuracy     : {test_acc_tree:.4f}")
print(f"  Overfitting gap   : {train_acc_tree - test_acc_tree:.4f}")
print()

# ---------------------------------------------------------------------------
# 3. BaggingClassifier with DecisionTreeStump (depth = 1)
# ---------------------------------------------------------------------------
bagging = BaggingClassifier(
    estimator=DecisionTreeClassifier(max_depth=1, random_state=42),
    n_estimators=100,
    max_samples=1.0,       # 100% of data (sampled with replacement)
    max_features=1.0,      # all features
    bootstrap=True,
    bootstrap_features=False,
    oob_score=True,        # enable out-of-bag score
    random_state=42
)
bagging.fit(X_train, y_train)

oob_score_bagging = bagging.oob_score_
test_acc_bagging = accuracy_score(y_test, bagging.predict(X_test))

print("=" * 60)
print("BAGGING (100 stumps, out-of-bag enabled)")
print(f"  OOB Score         : {oob_score_bagging:.4f}")
print(f"  Test accuracy     : {test_acc_bagging:.4f}")
print()

# ---------------------------------------------------------------------------
# 4. Bagging with DEEP decision tree (unpruned)
# ---------------------------------------------------------------------------
bagging_deep = BaggingClassifier(
    estimator=DecisionTreeClassifier(max_depth=None, random_state=42),
    n_estimators=100,
    max_samples=1.0,
    max_features=1.0,
    bootstrap=True,
    oob_score=True,
    random_state=42
)
bagging_deep.fit(X_train, y_train)

oob_deep = bagging_deep.oob_score_
test_deep = accuracy_score(y_test, bagging_deep.predict(X_test))

print("=" * 60)
print("BAGGING (100 deep trees, out-of-bag enabled)")
print(f"  OOB Score         : {oob_deep:.4f}")
print(f"  Test accuracy     : {test_deep:.4f}")
print()

# ---------------------------------------------------------------------------
# 5. Overfitting comparison
# ---------------------------------------------------------------------------
train_deep = accuracy_score(y_train, bagging_deep.predict(X_train))
gap_tree = train_acc_tree - test_acc_tree
gap_bagging = train_deep - test_deep

print("=" * 60)
print("OVERFITTING COMPARISON (train - test gap)")
print(f"  Single tree  : {gap_tree:.4f}")
print(f"  Bagging (deep) : {gap_bagging:.4f}")
print()

# Bagging significantly reduces overfitting because
# each tree sees a different subset of the data,
# and vote aggregation smooths out excessive predictions.

Results analysis

This code illustrates several important phenomena:

• The single tree typically shows strong overfitting: it achieves close to 100% accuracy on the training set, but its performance on the test set is significantly lower.
• Bagging with stumps (trees of depth 1) produces individually weak models, but their aggregation gives a solid result thanks to the diversity of subsamples.
• Bagging with deep trees is the most powerful configuration. Each tree is likely to overfit, but since each one overfits differently (on different data), the majority vote smooths out these excesses and produces a generalizable model.
• The out-of-bag score provides an honest estimate of generalized performance without requiring a separate validation set. It is calculated by evaluating each observation on the models that never saw it during training (the famous 37% excluded from the bootstrap).

Key Bagging Hyperparameters

The BaggingClassifier in scikit-learn exposes several essential hyperparameters that are critical to understand and tune properly.

n_estimators (integer, default: 10)

Number of models in the ensemble. The higher this number, the more variance is reduced. However, returns are diminishing: beyond 100 estimators, marginal gains often become negligible. In practice, it is recommended to start with 50 to 200 estimators and adjust according to the complexity of the problem.

max_samples (integer or float, default: 1.0)

Number or proportion of samples drawn for each individual training round. By default (max_samples=1.0), N samples are drawn with replacement (classic bootstrap). By reducing this value (e.g., max_samples=0.8), you increase diversity between models, which can be beneficial if the base models are too correlated. Caution: reducing max_samples too much can diminish the quality of each individual model.

max_features (integer or float, default: 1.0)

Number or proportion of features randomly drawn to train each model. By default, all features are used. By reducing max_features, you increase model diversity at the cost of a slight degradation of each individual model. This is the key parameter that distinguishes classic Bagging from Random Forests, where max_features is typically set to sqrt(n_features).

bootstrap (boolean, default: True)

Determines whether samples are drawn with replacement (True) or without replacement (False). Bootstrapping is the standard Bagging configuration. When bootstrap=False, it is sometimes called Pasting: each sample never contains the same observation twice. Pasting produces less diversified models than Bagging but can be useful when working with very large datasets.

random_state (integer, default: None)

Random seed for reproducibility. In research and production, it is strongly recommended to set this parameter to ensure that results are reproducible from one run to the next. Without random_state, each run will produce a different ensemble.

Advantages and Limitations of Bagging

Advantages

1. Effective variance reduction: Bagging significantly reduces variance without increasing bias, making it the ideal method for unstable models like deep decision trees.
2. No increased overfitting: Unlike Boosting, Bagging does not tend to overfit even with a very large number of estimators. Performance stabilizes and does not degrade.
3. Built-in evaluation (OOB): The out-of-bag score provides an estimate of generalized performance without requiring costly cross-validation. This is a considerable practical advantage.
4. Trivial parallelization: Each model is trained independently of the others. Bagging therefore parallelizes naturally across multiple processors or machines, significantly speeding up training.
5. Robustness to noise: Since each model only sees a subset of the data, noise present in certain observations has a limited impact on the final result. Noisy observations do not appear in all bootstrap samples.
6. Conceptual simplicity: The principle is simple to understand and implement. No notion of gradient, weights, or loss function is needed.

Limitations

1. Ineffective for stable models: If the base model is already stable (low variance), such as a linear regression or k-nearest neighbors with a large k, Bagging provides virtually no benefit. Correlated models do not diversify enough.
2. Limited bias reduction: Bagging only reduces variance, not bias. If the base model is biased (underfitting), Bagging will not solve the problem. In that case, Boosting (Adaptive Boosting) is more appropriate.
3. Loss of interpretability: An ensemble of 100 trees is much harder to interpret than a single tree. For applications requiring explainability (healthcare, regulated finance), Bagging can be problematic.
4. Computational cost: Training 100 models is 100 times more expensive than training one. Although parallelization mitigates this problem, the resources required remain significant for very large datasets.
5. No adaptive weighting: All models receive equal weight in the final vote. Particular models that make systematic errors on certain subspaces are not corrected in a targeted manner (unlike Boosting, which weights misclassified observations).

Four Concrete Use Cases for Bagging

Case 1: Banking Fraud Detection

In fraud detection, data is typically heavily imbalanced (99.9% legitimate transactions, 0.1% fraud). A single decision tree tends to ignore the minority class. Bagging makes it possible to combine several trees, each of which may have captured different aspects of fraudulent transactions through the diversity of bootstrap samples. Bagging can also be combined with undersampling of the majority class for each bootstrap, a technique known as Bagged Under Sampling.

Case 2: Medical Risk Prediction

In predictive medicine, robustness and reliability are essential. Bagging is particularly well suited because it produces more stable predictions than a single model. For example, to predict a patient’s risk of hospital readmission, a bagged ensemble provides predictions that are less sensitive to the specificities of a particular sample. The out-of-bag score also offers built-in validation, valuable when medical datasets are limited in size.

Case 3: Mortgage Credit Assessment

Financial institutions make extensive use of ensemble methods to assess borrower creditworthiness. Bagging reduces the variance of credit decisions, resulting in a lower and more consistent error rate. In a highly regulated context, being able to estimate prediction confidence via the out-of-bag score is a significant regulatory advantage.

Case 4: Industrial Predictive Maintenance

In Industry 4.0, IoT sensors generate massive streams of telemetry data. Bagging is ideal for building failure prediction models: each estimator can capture different failure patterns (abnormal vibrations, temperature rise, progressive wear), and vote aggregation reduces false positives. The native parallelization of Bagging also allows these models to be trained on distributed clusters, which is essential given the volume of industrial data.

Conclusion

Bagging remains one of the most elegant and useful ensemble techniques in machine learning. Its fundamental principle — training multiple models on bootstrap subsamples and aggregating their predictions — is remarkably simple, but its theoretical implications are profound. By reducing variance by a factor of up to k (the number of models), Bagging transforms unstable models into robust and reliable predictions. Whether used alone with a BaggingClassifier or as the foundation of Random Forests, Bagging is an indispensable tool in any machine learning practitioner’s toolkit.

See Also

• Mastering Reflexive Position in Python: Complete Guide for Developers
• Euclidean Algorithm in Python: Efficiently Computing the Greatest Common Divisor