Decision Tree (Regression): Complete Guide — Principles, Examples, and Python Implementation

Summary

The decision tree regression method is a supervised learning approach that recursively partitions the feature space into disjoint regions, then assigns to each region the mean of the target values of the training points that fall within it. Unlike the classification tree which uses entropy or the Gini index, the regression tree relies on variance reduction (or Mean Squared Error, MSE) to choose its split points. This approach produces a step function, capable of capturing complex non-linear relationships without a priori distributional assumptions. This guide covers the mathematical principle, geometric intuition, Python implementation with scikit-learn, key hyperparameters, and concrete use cases.

Mathematical Principle

Data and Objective

We have a training set {(x₁, y₁), …, (xₙ, yₙ)} where xᵢ is a feature vector and yᵢ is a continuous target. The objective is to construct a function f(x) that approximates y with minimal squared error.

MSE Criterion (Mean Squared Error)

The CART algorithm (Classification And Regression Trees) evaluates each potential split according to the mean squared error reduction:

MSE(S) = (1 / |S|) × Σᵢ∈ₛ (yᵢ – ȳ_S)²

where ȳ_S is the mean of the targets in subset S.

Choosing the Best Split

For each feature j and each candidate threshold s, the current node is split into two subsets:

• S_left = {i | x_ij ≤ s}
• S_right = {i | x_ij > s}

The split cost is the weighted sum of the MSE of the two children:

Cost(S_left, S_right) = (|S_left| / |S|) × MSE(S_left) + (|S_right| / |S|) × MSE(S_right)

The algorithm systematically tests all variables and all unique observed thresholds, then retains the pair (j, s) that minimizes this cost. This greedy strategy guarantees a maximum local variance reduction at each step.

Leaf Prediction

Once the terminal leaf is reached, the prediction is simply:

f(x) = ȳ_leaf = (1 / |S_leaf|) × Σ of yᵢ in the leaf

This mean constitutes the unbiased estimator of the target in the region covered by the leaf.

Complexity and Stopping

The tree construction continues until a stopping criterion is reached: maximum depth, minimum number of samples in a node, or insufficient MSE reduction. Without regularization, the tree can grow until each leaf contains a single sample, which corresponds to perfect overfitting (zero MSE on training, catastrophic performance on new data).

Geometric Intuition

The Tree as a Space Partition

Imagine you have a two-dimensional point cloud with a continuous target variable. The decision tree regression cuts this space with axis-parallel lines. Each cut is a binary question of the form “Is feature X₁ less than 3.7?” The result is a rectangular tessellation of the input space.

Each Leaf = Local Mean

In each terminal rectangle, the tree predicts the mean of the y values of the training points within it. Graphically, this amounts to replacing the point cloud with a staircase surface: flat in each region, with discontinuities at the leaf boundaries.

Depth vs. Granularity

• Shallow tree (max_depth = 2-3): a few large regions, smoothed predictions, high bias but low variance.
• Deep tree (max_depth = 10+): many small regions, very fine predictions, low bias but risk of overfitting.

The fundamental paradox is: a tree that is too simple does not capture local data variations, while a tree that is too complex learns the noise. Pruning seeks the equilibrium point between these two extremes.

Analogy with Histograms

The one-dimensional regression tree is conceptually close to an adaptive histogram: instead of dividing the axis into fixed-width bins, the algorithm places cuts where the intra-bin variance is maximized. In multiple dimensions, this idea generalizes by cutting along different variables at each level.

Python Implementation with scikit-learn

Fundamental Example: Noisy Sinusoid

import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split

# Data generation: sinusoid with Gaussian noise
rng = np.random.RandomState(42)
X = np.sort(5 * rng.rand(200, 1), axis=0)
y = np.sin(X).ravel() + 0.3 * rng.randn(200)

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# Fit regression tree
regressor = DecisionTreeRegressor(max_depth=4, random_state=42)
regressor.fit(X_train, y_train)

# Predictions and evaluation
train_score = regressor.score(X_train, y_train)
test_score = regressor.score(X_test, y_test)
print(f"R² training : {train_score:.4f}")
print(f"R² test : {test_score:.4f}")

Visualizing the Impact of Depth

# Comparison of multiple depths
plt.figure(figsize=(14, 6))

X_plot = np.linspace(0, 5, 500).reshape(-1, 1)

for i, depth in enumerate([1, 2, 4, 10], 1):
    model = DecisionTreeRegressor(max_depth=depth, random_state=42)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_plot)

    plt.subplot(2, 2, i)
    plt.scatter(X_train, y_train, c="darkorange", alpha=0.4, label="Data")
    plt.plot(X_plot, y_pred, c="navy", linewidth=2, label=f"max_depth={depth}")
    plt.title(f"Depth = {depth} (R²={model.score(X_test, y_test):.3f})")
    plt.xlabel("x")
    plt.ylabel("y")
    plt.legend()

plt.tight_layout()
plt.savefig("decision_tree_regression_depth_comparison.png", dpi=150)
plt.close()

The results perfectly illustrate the bias-variance tradeoff:

• max_depth = 1: a single cut, the model is a two-step staircase. It massively underfits and does not capture the sinusoidal curve.
• max_depth = 2: three approximate steps. The model starts to follow the general trend but misses details.
• max_depth = 4: sixteen steps. The fit is good, test R² is close to training R². This is often the optimal zone.
• max_depth = 10: the tree can have up to 1024 leaves. The model fits the noise, creating an irregular staircase with many spurious steps. Training R² approaches 1.0, but test R² drops — a sign of overfitting.

Cost-Complexity Pruning (CCP)

from sklearn.tree import DecisionTreeRegressor

# Fit the complete tree without restrictions
regressor_full = DecisionTreeRegressor(random_state=42)
regressor_full.fit(X_train, y_train)

# Extract the CCP regularization path
path = regressor_full.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas = path.ccp_alphas

# Train a series of pruned trees
models = []
for alpha in ccp_alphas[:-1]:
    clf = DecisionTreeRegressor(ccp_alpha=alpha, random_state=42)
    clf.fit(X_train, y_train)
    models.append((alpha, clf))

# Select the best alpha according to test score
best_alpha, best_model = max(models, key=lambda m: m[1].score(X_test, y_test))
print(f"Best ccp_alpha : {best_alpha:.6f}")
print(f"Test R² of pruned model : {best_model.score(X_test, y_test):.4f}")
print(f"Number of leaves : {best_model.get_n_leaves()}")
print(f"Effective depth : {best_model.get_depth()}")

CCP pruning is preferable to directly limiting max_depth because it operates an optimal tradeoff between complexity (number of leaves) and training performance, based on a mathematically founded criterion.

Key Hyperparameters

criterion

• "squared_error" (default): minimizes intra-leaf variance (MSE).
• "friedman_mse": variant proposed by Jerome Friedman. Can give better splits in some cases.
• "absolute_error": minimizes MAE. More robust to outliers.
• "poisson": for non-negative count data following a Poisson distribution. Useful in actuarial science.

max_depth

Maximum depth directly controls the model’s expressive capacity. A tree of depth d can have up to 2ᵈ leaves. In practice, values between 3 and 8 are frequently optimal. Beyond 15, overfitting becomes virtually inevitable without other regularization mechanisms.

min_samples_split

The minimum number of samples required to split an internal node. The default value is 2. Increasing this parameter (to 5, 10, or 20) reduces complexity by preventing splits on subsets that are too small.

min_samples_leaf

The minimum number of samples that each terminal leaf must contain. This is one of the most effective hyperparameters for controlling overfitting. A value of 5 to 10 is a good starting point.

max_features

The number of features considered at each split. By default, all variables are examined. Reducing this number introduces randomness, used in random forests.

ccp_alpha

The regularization parameter for cost-complexity pruning. Each added leaf costs ccp_alpha in penalty. Cross-validation on a grid of logarithmic values is the recommended method for selecting this parameter.

Advantages and Limitations

Advantages of Decision Tree Regression

1. No distributional assumption: the algorithm assumes neither linearity, nor normality of residuals, nor homoscedasticity. It adapts to the shape of the data.
2. Invariance to monotonic transformations: applying a logarithm or rank to a variable does not affect possible splits, since only the order of values matters.
3. Native handling of mixed variables: numerical and categorical variables coexist without prior encoding.
4. Interpretability: the tree structure is readable by a human, unlike deep neural networks.
5. Fast inference: a prediction requires only d comparisons, where d is the depth. This is extremely fast in production.
6. Robustness to missing values: scikit-learn natively handles NaNs since version 1.1.

Limitations of Decision Tree Regression

1. Instability: a small change in the data can produce a radically different tree. This is the raison d’être of bagging and random forests.
2. Underfitting of linear trends: a tree approximates a straight line with a staircase. For linear relationships, a linear regression is more efficient.
3. Impossible extrapolation: the tree cannot predict outside the range observed during training.
4. Difficulty with diagonal correlations: axis-parallel cuts require many splits to approximate an oblique boundary.
5. Bias toward high-cardinality variables: a continuous variable with many values has more split candidates, biasing the selection.
6. Staircase predictions: the decision function is discontinuous, which can be unphysical (temperature, pressure).

4 Concrete Use Cases

Case 1 — Real Estate Estimation

A regression tree can estimate the price of a property based on its area, number of rooms, year of construction, neighborhood, and distance to transportation. The tree structure produces explicit rules: for example, “Area > 80m² AND downtown neighborhood AND year > 2000 → average price = 450,000 euros.” This transparency is a major asset for buyers and regulators. To improve accuracy, several trees are typically combined in a Gradient Boosting (XGBoost, LightGBM).

Case 2 — Energy Demand Forecasting

Electrical grid managers forecast hourly consumption based on outside temperature, day of the week, time of day, and holidays. The regression tree naturally captures threshold effects: for example, consumption explodes below 5°C (heating) and above 30°C (air conditioning). These abrupt transitions are ideally modeled by tree splits. Fast prediction enables real-time production adjustment.

Case 3 — Credit Risk Assessment

In credit scoring, the regression tree evaluates borrower default probability based on income, indebtedness, payment history, and employment seniority. The decision rules from the tree are directly usable to justify a loan refusal, which is a regulatory requirement in many jurisdictions (GDPR right to explanation in Europe).

Case 4 — Industrial Sensor Calibration

In industry, inexpensive sensors present non-linear distortions compared to reference sensors. A regression tree trained on pairs (inexpensive sensor reading, reference sensor reading) learns a piecewise correction function. Each leaf corresponds to a range of raw values, and the prediction is the mean bias in that range. This approach is simple, deterministic, and deployable in resource-constrained embedded microcontrollers.

See Also

• Master Mahjong with Python: Complete Guide for Developers and Enthusiasts
• Mastering Grid Graphs with Python: Complete Guide and Optimization Tips