Quantile Regression: Complete Guide — Régression Quantile

Summary — Quantile regression is a regression technique that predicts conditional quantiles (not just the mean) of the target variable distribution. Introduced by Koenker and Bassett in 1978, it is particularly useful when the error distribution is not symmetric or when you are interested in the extremes (risks, unusual values). Unlike least squares regression which gives a single symmetric point prediction, quantile regression allows you to build asymmetric prediction intervals adapted to the reality of the data.

Mathematical Principle

1. Limitations of Ordinary Least Squares (OLS) Regression

Classical linear regression minimizes the sum of squared residuals and predicts the conditional mean E[Y|X=x]. The problem: it assumes errors are symmetric and homoscedastic. In reality, distributions are often asymmetric (salaries, commute times, real estate prices).

2. The check loss function (pinball loss)

To predict the tau-th conditional quantile Q_tau(Y|X=x), we minimize the check loss:

rho_tau(u) = tau · max(u, 0) + (1-tau) · max(-u, 0)
           = max(tau · u, (tau-1) · u)

Where u = y – f(x) is the residual and tau is in ]0, 1[. The tau quantile represents the value below which tau × 100% of observations fall.

For tau = 0.5 (median):

rho_0.5(u) = 0.5 · |u|  →  this is equivalent to minimizing MAE

For tau = 0.9 (90th percentile):

rho_0.9(u) = 0.9 · max(u, 0) + 0.1 · max(-u, 0)

Positive errors (under-prediction) are penalized 9x more than negative errors (over-prediction), which pushes the model toward the upper part of the distribution.

3. Linear Quantile Regression

The linear quantile model estimates:

Q_tau(Y|X=x) = x^T · beta_tau
beta_tau = argmin_beta sum_i rho_tau(y_i - x_i^T · beta)

Note that the coefficients beta_tau depend on tau: the 90th percentile regression gives different weights than the 10th percentile regression. This is fundamental: each quantile captures a different relationship between X and Y.

4. Prediction Intervals

To obtain a 90% prediction interval, you train two models:
– tau = 0.05 → lower bound
– tau = 0.95 → upper bound

The interval [Q_0.05(Y|x), Q_0.95(Y|x)] contains 90% of the conditional distribution of Y given X=x.

5. Non-linear Quantile Regression (Deep Quantile Regression)

You can replace the linear model with a neural network:

Q_tau(Y|X=x) = f_theta(x)  where f is an MLP
theta_tau = argmin_theta sum_i rho_tau(y_i - f_theta(x_i))

An advantage: you can train a single network to predict multiple quantiles simultaneously by having a last layer with K outputs (one per quantile) and a total loss:

L = sum_k sum_i rho_{tau_k}(y_i - f_theta(x_i)_k)

Intuition

Classical regression tells you: “the commute will take 30 minutes on average.” But in reality, sometimes it’s 20 minutes, sometimes 50. Quantile regression tells you: “in 90% of cases, the commute lasts between 22 and 42 minutes.”

This is the difference between a weather report giving the average temperature and one giving the min and max — the second one really helps you decide what to wear.

Python Implementation

1. With scikit-learn

import numpy as np
from sklearn.linear_model import QuantileRegressor
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

# Synthetic data with heteroscedasticity
np.random.seed(42)
n = 200
X = np.random.uniform(0, 10, n).reshape(-1, 1)
y = 2 * X.ravel() + np.random.normal(0, X.ravel(), n)  # variance increases with X

# Regression on 3 quantiles
quantiles = [0.1, 0.5, 0.9]
fits = {}

for q in quantiles:
    qr = QuantileRegressor(quantile=q, alpha=0.1, solver='highs')
    qr.fit(X, y)
    fits[q] = qr

X_plot = np.linspace(0, 10, 100).reshape(-1, 1)

fig, ax = plt.subplots(figsize=(10, 6))
ax.scatter(X, y, alpha=0.5, label='Data', c='blue')

colors = ['red', 'green', 'red']
linestyles = ['--', '-', '--']
for i, q in enumerate(quantiles):
    y_pred = fits[q].predict(X_plot)
    ax.plot(X_plot, y_pred, color=colors[i], linestyle=linestyles[i],
            label=f'Quantile {q}')

# 80% interval zone
ax.fill_between(X_plot.ravel(),
                fits[0.1].predict(X_plot),
                fits[0.9].predict(X_plot),
                alpha=0.15, color='green', label='80% Interval')

ax.set_xlabel('X'); ax.set_ylabel('y')
ax.set_title('Quantile Regression on heteroscedastic data')
ax.legend()
plt.tight_layout()
plt.savefig('quantile_regression.png', dpi=150)

2. Deep Quantile Regression with PyTorch

import torch
import torch.nn as nn
import numpy as np

def pinball_loss(pred, target, tau):
    """Pinball loss function for quantile regression."""
    residuals = target - pred
    return torch.mean(torch.max(tau * residuals, (tau - 1) * residuals))

class QuantileNet(nn.Module):
    """Neural network predicting multiple quantiles simultaneously."""
    def __init__(self, input_dim=1, n_quantiles=3):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(input_dim, 64), nn.ReLU(),
            nn.Linear(64, 64), nn.ReLU(),
            nn.Linear(64, n_quantiles)  # One output per quantile
        )
        self.quantiles = torch.linspace(0.1, 0.9, n_quantiles)

    def forward(self, x):
        return self.network(x)

# Data
X = torch.randn(1000, 1)
y = 3 * X + torch.randn(1000, 1) * (1 + 0.5 * torch.abs(X))

# Training
model = QuantileNet(input_dim=1, n_quantiles=3)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for epoch in range(200):
    preds = model(X)  # batch x n_quantiles
    loss = 0
    for k, tau in enumerate(model.quantiles):
        loss += pinball_loss(preds[:, k], y.squeeze(), tau)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if epoch % 50 == 0:
        print(f'Epoch {epoch} | Loss: {loss.item():.4f}')

# Predictions
X_test = torch.linspace(-3, 3, 100).reshape(-1, 1)
with torch.no_grad():
    preds = model(X_test)
    for k, tau in enumerate(model.quantiles):
        q_val = f'{tau:.1f}'
        print('Quantile ' + str(q_val) + ': ' + str(preds[50, k].item()))

Hyperparameters

Advantages

1. No normality assumption: Unlike OLS, quantile regression does not assume Gaussian errors. It works with any distribution.
2. Robust to outliers: The median (tau=0.5) is much more robust to extreme values than the mean. A single outlier does not change the median but can shift the mean significantly.
3. Complete view of the distribution: Predicting multiple quantiles gives a complete picture of how Y varies as a function of X, including in the tails of the distribution.
4. Direct interpretation: The interval [Q_0.05, Q_0.95] has a natural frequentist interpretation: 90% of observations fall within it.

Limitations

1. Quantile crossing: The predicted quantiles can cross (Q_0.9 < Q_0.1) for certain values of X, which is inconsistent. Solutions exist (order constraints, non-crossing quantile regression).
2. Computational cost: Optimizing the pinball loss is a linear program (no closed-form solution like OLS). For large datasets, it is slower.
3. Less interpretable in high dimensions: With many features, understanding the coefficients of each quantile becomes complex.

Comparison with other uncertainty methods

Unlike Bayesian methods (which model uncertainty on parameters) or model ensembles (which average multiple predictions), quantile regression is purely frequentist: it directly estimates the empirical quantiles of the distribution. It is simpler to implement than GPs and more interpretable than deep network ensembles.

4 Concrete Use Cases

1. Energy Demand Forecasting

Electricity companies predict demand with quantile intervals. The 0.95 quantile ensures there is enough capacity to cover 95% of scenarios — essential to avoid blackouts.

2. Financial Risk Assessment (Value at Risk)

VaR (Value at Risk) is essentially a quantile (usually tau=0.01 or 0.05). Quantile regression allows predicting conditional VaR as a function of market variables, adapting risk to the context.

3. Medicine: Pediatric Growth Curves

WHO growth charts are based on percentiles (quantiles). Quantile regression makes it possible to model the distribution of height/weight as a function of age, sex, and other covariates.

4. Meteorology: Probabilistic Forecasting

Weather services use quantile regression to produce forecasts with confidence intervals, allowing farmers and planners to assess the risks of drought or flooding.

Conclusion

Quantile regression is a fundamental tool for anyone who wants to go beyond the mean and understand the complete distribution of conditional data. From economics to medicine to meteorology, it provides informative prediction intervals where a single point simply isn’t enough.

Modern extensions (quantile neural networks, quantile crossing avoidance) extend its applicability to high-dimensional problems and complex non-linearity.

See Also

• Deux modules pour bien sécuriser votre navigateur web
• Tester la Primalité de $2n^2 – 1$ en Python : Guide Complet et Optimisé