Decision Stump: Complete Guide — Principles, Examples, and Python Implementation

Summary — The Decision Stump (or “decision tree with a single branch”) is the simplest possible classifier: it asks ONE question about ONE feature and makes its decision accordingly. Considered individually, it is a “weak learner” with accuracy barely above chance. But assembled by the thousands in boosting algorithms like AdaBoost or Gradient Boosting, it becomes a fundamental building block of modern machine learning.

Mathematical Principle

The Decision Stump is a decision tree of depth 1: exactly one decision node (split) and two leaves.

1. Model Structure

For each feature j in the dataset, and for each possible threshold t among the values taken by that feature:

• If x[j] ≤ t then predict the majority class on the left
• Else predict the majority class on the right

The optimal (feature, threshold) pair is the one that minimizes the total impurity of the two child nodes.

2. Impurity Measures

Two criteria are primarily used to evaluate the quality of a split:

Gini Index:
$$G = 1 – \sum_{k=1}^{K} p_k^2$$

where p_k is the proportion of examples of class k in the node. For a binary problem (classes +1 and -1):
$$G = 1 – p_+^2 – p_-^2 = 2 \cdot p_+ \cdot (1 – p_+)$$

A pure node (single class) has a Gini of 0. A perfectly mixed node (50/50) has a Gini of 0.5.

Entropy (Information Gain):
$$H = -\sum_{k=1}^{K} p_k \cdot \log_2(p_k)$$

For a binary node: H = -p₊·log₂(p₊) – p₋·log₂(p₋). Entropy is maximal (1.0) when classes are perfectly mixed and zero when the node is pure.

3. Optimal Split Criterion

For each feature j and each candidate threshold t, we compute the weighted impurity of the two child nodes:

$$I(j, t) = \frac{N_G}{N} \cdot \text{Impurity}(G) + \frac{N_D}{N} \cdot \text{Impurity}(D)$$

where N_G and N_D are the number of examples to the left and right of the threshold, and N = N_G + N_D is the total number of examples.

The best split is the one that minimizes this weighted impurity:

$$(j^, t^) = \arg\min_{j, t} I(j, t)$$

This is equivalent to maximizing the information gain (impurity reduction):

$$\Delta I = \text{Impurity}_{parent} – I(j^, t^)$$

4. Post-Split Prediction

After finding the optimal split, each leaf predicts the majority class of the examples that fall into it:

• Left leaf (x[j] ≤ t): class y_G = mode({y_i | x_i[j] ≤ t})
• Right leaf (x[j] > t): class y_D = mode({y_i | x_i[j] > t})

Intuition

The Decision Stump asks ONE question and makes its decision accordingly.

Imagine you need to guess whether a person is going to buy a product on a website. You have thousands of features at your disposal: age, location, purchase history, time spent on the site, etc.

The Decision Stump will only ask one question:

“Is the time spent on the site greater than 30 seconds?”

• Yes → Predict: likely purchase
• No → Predict: no purchase

That’s it. One test, two branches, two predictions.

This seems too simple to be useful, and it’s true: a Decision Stump alone is a very mediocre classifier, barely better than chance. Its accuracy rarely exceeds 60-70%, even on simple problems.

But — and this is the magic of boosting — if you assemble hundreds or thousands of Decision Stumps, each asking its own question about its own feature, and you combine them through weighted voting, you get an extremely powerful classifier. This is exactly the principle of AdaBoost: each stump is an imperfect expert, and the ensemble is a council of experts whose collective wisdom far exceeds that of any individual.

Python Implementation

Example 1: Decision Stump from Scratch

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

class DecisionStump:
    """Decision Stump - un arbre de décision de profondeur 1."""

    def __init__(self, criterion='gini'):
        self.criterion = criterion
        self.best_feature = None
        self.best_threshold = None
        self.left_class = None
        self.right_class = None

    def _gini(self, y):
        if len(y) == 0:
            return 0.0
        p = np.mean(y == 1)
        return 2 * p * (1 - p)

    def _entropy(self, y):
        if len(y) == 0:
            return 0.0
        p = np.mean(y == 1)
        if p == 0 or p == 1:
            return 0.0
        return -(p * np.log2(p) + (1 - p) * np.log2(1 - p))

    def _impurity(self, y):
        if self.criterion == 'gini':
            return self._gini(y)
        else:
            return self._entropy(y)

    def fit(self, X, y):
        n_samples, n_features = X.shape
        best_impurity = float('inf')

        for feature_idx in range(n_features):
            # Candidate thresholds: medians between sorted unique values
            thresholds = np.unique(X[:, feature_idx])
            if len(thresholds) > 50:
                thresholds = np.percentile(X[:, feature_idx],
                    np.linspace(0, 100, 50))

            for threshold in thresholds:
                left_mask = X[:, feature_idx] <= threshold
                right_mask = ~left_mask

                if np.sum(left_mask) == 0 or np.sum(right_mask) == 0:
                    continue

                left_impurity = self._impurity(y[left_mask])
                right_impurity = self._impurity(y[right_mask])

                n_left = np.sum(left_mask)
                n_right = np.sum(right_mask)
                weighted_impurity = (n_left * left_impurity +
                                    n_right * right_impurity) / n_samples

                if weighted_impurity < best_impurity:
                    best_impurity = weighted_impurity
                    self.best_feature = feature_idx
                    self.best_threshold = threshold
                    self.left_class = 1 if np.mean(y[left_mask]) > 0.5 else 0
                    self.right_class = 1 if np.mean(y[right_mask]) > 0.5 else 0

        return self

    def predict(self, X):
        predictions = np.where(
            X[:, self.best_feature] <= self.best_threshold,
            self.left_class, self.right_class
        )
        return predictions

# Data
X, y = make_classification(n_samples=500, n_features=10, n_informative=3,
    n_redundant=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
    random_state=42)

# Training
stump_gini = DecisionStump(criterion='gini')
stump_gini.fit(X_train, y_train)
pred_gini = stump_gini.predict(X_test)

stump_ent = DecisionStump(criterion='entropy')
stump_ent.fit(X_train, y_train)
pred_ent = stump_ent.predict(X_test)

print(f"Stump (Gini)    : {accuracy_score(y_test, pred_gini):.3f}")
print(f"Stump (Entropy) : {accuracy_score(y_test, pred_ent):.3f}")
print(f"Feature used     : {stump_gini.best_feature}")
print(f"Optimal threshold: {stump_gini.best_threshold:.4f}")

Example 2: Comparison with DecisionTreeClassifier(max_depth=1)

from sklearn.tree import DecisionTreeClassifier

# sklearn Decision Stump (depth-1 tree)
sklearn_stump = DecisionTreeClassifier(max_depth=1, random_state=42)
sklearn_stump.fit(X_train, y_train)
sklearn_pred = sklearn_stump.predict(X_test)

print(f"\nComparison:")
print(f"Stump from scratch (Gini)  : {accuracy_score(y_test, pred_gini):.3f}")
print(f"sklearn DecisionTree(d=1)  : {accuracy_score(y_test, sklearn_pred):.3f}")

# Boundary visualization
import matplotlib.pyplot as plt

# Use 2 features for visualization
X2 = X[:, :2]
X2_train, X2_test, y2_train, y2_test = train_test_split(X2, y, test_size=0.3,
    random_state=42)

stump2 = DecisionStump()
stump2.fit(X2_train, y2_train)

fig, ax = plt.subplots(figsize=(8, 6))
xx, yy = np.meshgrid(np.linspace(X2[:, 0].min()-1, X2[:, 0].max()+1, 200),
                     np.linspace(X2[:, 1].min()-1, X2[:, 1].max()+1, 200))
Z = stump2.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
ax.contourf(xx, yy, Z, alpha=0.3, cmap='RdBu')
ax.scatter(X2_train[:, 0], X2_train[:, 1], c=y2_train, cmap='RdBu',
    edgecolors='black', s=30)

# Draw the boundary
f = stump2.best_feature
t = stump2.best_threshold
if f == 0:
    ax.axvline(x=t, color='black', linestyle='--', linewidth=2)
else:
    ax.axhline(y=t, color='black', linestyle='--', linewidth=2)

ax.set_title(f'Decision Stump — Feature {f}, Threshold = {t:.3f}')
plt.tight_layout()
plt.savefig('decision_stump_boundary.png', dpi=150)

Example 3: Decision Stump as AdaBoost Base

from sklearn.ensemble import AdaBoostClassifier

# AdaBoost with Decision Stumps (max_depth=1 by default)
ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),
    n_estimators=200,
    learning_rate=1.0,
    random_state=42
)
ada.fit(X_train, y_train)
ada_pred = ada.predict(X_test)

print(f"\nAdaBoost (200 stumps) : {accuracy_score(y_test, ada_pred):.3f}")
print(f"A single stump alone   : {accuracy_score(y_test, pred_gini):.3f}")
print(f"Performance gain       : {accuracy_score(y_test, ada_pred) - accuracy_score(y_test, pred_gini):.3f}")

# Cumulative score curve visualization
scores = []
for n in range(1, 201, 10):
    ada_n = AdaBoostClassifier(
        estimator=DecisionTreeClassifier(max_depth=1),
        n_estimators=n, learning_rate=1.0, random_state=42
    )
    ada_n.fit(X_train, y_train)
    scores.append(accuracy_score(y_test, ada_n.predict(X_test)))

plt.figure(figsize=(8, 5))
plt.plot(range(1, 201, 10), scores, 'b-o', linewidth=2, markersize=4)
plt.axhline(y=accuracy_score(y_test, pred_gini), color='red', linestyle='--',
    label='Single stump')
plt.title('AdaBoost — Accumulation of Decision Stumps')
plt.xlabel('Number of stumps')
plt.ylabel('Accuracy')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('adaboost_stumps.png', dpi=150)

Example 4: Analysis of AdaBoost Weights

# Which stumps are most important in AdaBoost?
ada_full = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),
    n_estimators=50, learning_rate=1.0, random_state=42
)
ada_full.fit(X_train, y_train)

print("\nWeights of the first stumps in AdaBoost:")
for i, (tree, w) in enumerate(zip(ada_full.estimators_, ada_full.estimator_weights_)):
    feat = tree.tree_.feature[0]
    thresh = tree.tree_.threshold[0]
    print(f"  Stump {i+1}: feature {feat}, threshold {thresh:.3f}, weight {w:.4f}")

Hyperparameters

Advantages of the Decision Stump

1. Extreme simplicity: One test, two branches. This is the simplest possible model in supervised classification, making it perfectly interpretable.
2. Training speed: Finding the best split is O(n_features × n_unique_values × n_samples), very fast compared to a complete tree.
3. Fundamental building block of boosting: AdaBoost, Gradient Boosting, and XGBoost all use Decision Stumps as base weak learners. Understanding the stump means understanding the elementary brick of these powerful algorithms.
4. Total interpretability: The decision rule is a single condition readable by a human. This is a major asset in regulated domains (finance, healthcare) where every decision must be explainable.
5. Not very prone to individual overfitting: A Decision Stump alone has high bias but very low variance — it never overfits. This property is what makes it so useful in ensembles.

Limitations of the Decision Stump

1. Very limited individual performance: A Decision Stump alone rarely achieves more than 60-70% accuracy. This is insufficient for most real-world applications without boosting.
2. Very simple decision boundary: The boundary is always a straight line parallel to an axis. Impossible to capture nonlinear relationships or feature interactions.
3. Instability of the optimal split: A slight change in training data can select a completely different feature or threshold. The individual stump is unstable.
4. Inability to model interactions: Since it only tests one feature, a Decision Stump is blind to feature combinations (e.g., “age < 30 AND income > 50,000”).
5. Dependence on impurity criterion: The choice between Gini and Entropy can produce different splits, although the end results are generally similar in practice.

4 Concrete Use Cases

1. Spam Detection (Low False Alarm Cost)

A Decision Stump can detect spam with a very simple single rule: “Does the word ‘free’ appear more than 3 times in the email?” Although imperfect, this quick filter eliminates a significant fraction of obvious spam before passing the rest to a more sophisticated classifier.

2. Medical Risk Prediction (Interpretable Model)

In a clinical context where interpretability is crucial, a Decision Stump can offer a simple triage rule: “Is the fasting blood glucose greater than 126 mg/dL?” This single rule, while not exhaustive, can serve as a first alert filter before additional analysis.

3. AdaBoost Base Building Block in Production

This is the primary use of the Decision Stump in practice. Credit scoring, churn prediction, and recommendation systems often use ensembles of stumps via Gradient Boosting (XGBoost, LightGBM). Each stump is simple, but thousands combined produce models competitive with neural networks on tabular data.

4. Quick Feature Selection

By training a Decision Stump on each feature individually and comparing their performances, you get a quick ranking of features by individual discriminating power. This is a very simple but effective feature selection method for identifying the most informative variables before training more complex models.

Conclusion

The Decision Stump is the humblest classifier in machine learning: one question, two branches, two answers. Individually, it is so weak that it is called a “weak learner.” It is precisely this weakness that makes it so powerful in boosting, where thousands of stumps assembled together give rise to some of the most performant models for tabular data.

Understanding the Decision Stump means understanding the elementary brick of AdaBoost, Gradient Boosting, XGBoost, and LightGBM — algorithms that dominate Kaggle competitions and industrial deployments on structured data.

See Also

• Master Pandigital Concatenated Products with Python: Complete Guide for Developers in 2024
• Create Unique Sum Subsets in Python: Complete Guide and Advanced Tips