Linear Regression: Principles, Examples, and Python Implementation

Mathematical Principle

Linear regression is based on a surprisingly simple starting idea: assuming that there exists an approximately linear relationship between the input variables (called features) and the target variable we are trying to predict.

The Model

The fundamental equation of linear regression is written:

$$y = w_0 + w_1 x_1 + w_2 x_2 + \cdots + w_p x_p + \varepsilon$$

where:

• $y$ is the target variable (or dependent variable), the one we want to predict.
• $x_1, x_2, \ldots, x_p$ are the explanatory variables (or features), numbering $p$.
• $w_0$ is the intercept, also called the bias. It is the value of $y$ when all features are equal to zero.
• $w_1, w_2, \ldots, w_p$ are the coefficients (or weights) associated with each feature. Each $w_j$ measures the impact of $x_j$ on the target variable, all other things being equal.
• $\varepsilon$ is the error term (or noise), capturing everything the model does not explain: random variations, nonlinearities not captured, measurement noise.

Cost Function: Mean Squared Error (MSE)

To find the best coefficients $w_0, w_1, \ldots, w_p$, we define a cost function that measures the quality of our predictions. The most commonly used cost function in linear regression is the Mean Squared Error (MSE):

$$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i – \hat{y}_i)^2$$

where:

• $n$ is the total number of samples in our training data.
• $y_i$ is the actual observed value for the $i$-th sample.
• $\hat{y}_i$ is the value predicted by our model for the $i$-th sample.
• $(y_i – \hat{y}_i)$ is the residual, i.e., the error made on the $i$-th sample.

Squaring the residuals penalizes large errors more than small ones, making the model sensitive to outliers.

Analytical Solution: The Normal Equation

The beauty of linear regression lies in the fact that it admits a closed-form analytical solution. By differentiating the MSE cost function with respect to the coefficients and setting this derivative to zero, we obtain the normal equation:

$$\hat{w} = (X^\top X)^{-1} X^\top y$$

where:

• $\hat{w}$ is the vector of optimal coefficients $[w_0, w_1, \ldots, w_p]^\top$.
• $X$ is the design matrix of size $n \times (p+1)$, where each row corresponds to a sample and the first column is typically a column of 1s corresponding to the bias $w_0$.
• $X^\top$ denotes the transpose of matrix $X$.
• $y$ is the column vector of target values of size $n \times 1$.
• $(X^\top X)^{-1}$ is the inverse of the matrix $X^\top X$. This matrix must be invertible for the solution to exist.

This solution is exact and requires no iteration. However, computing the inverse of a large matrix can be computationally expensive for very large datasets (above approximately 100,000 samples), which justifies the use of iterative methods like gradient descent in those situations.

Intuition — How to Understand It?

The Geometric Analogy

Imagine a scatter plot drawn on a Cartesian plane: each point represents an observation (for example, the area of an apartment on the x-axis and its price on the y-axis). Linear regression consists of finding the line that best passes through this cloud of points.

This « optimal » line is the one that minimizes the sum of the vertical distances between each actual point and the line itself. These vertical distances are precisely the residuals mentioned above.

Why Vertical Distances?

One might imagine measuring the shortest perpendicular distance between the point and the line, but in classical linear regression, we are only interested in the error on the target variable $y$. This is why we minimize the vertical gaps (on the $y$-axis) rather than the perpendicular distances.

The Carpet Analogy

Imagine you need to lay a rigid carpet (your regression line) in a room where nails are driven into the wall at different heights (your data). You want the carpet to pass as close as possible to each nail. The best position is the one where the sum of the gaps between the carpet and each nail is minimal. That is exactly what linear regression does.

Python Implementation

Installing Dependencies

Let’s start by installing the necessary libraries:

pip install scikit-learn numpy matplotlib

Complete Code

Here is a complete, executable example that illustrates all the steps of linear regression:

# -*- coding: utf-8 -*-
"""
Complete example of linear regression with scikit-learn.
Synthetic data, training, evaluation, and visualization.
"""

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# -----------------------------------------------------------
# Step 1: Generating synthetic data
# -----------------------------------------------------------
# We create a dataset with 200 samples and
# a single explanatory variable for clear visualization.
X, y = make_regression(
    n_samples=200,
    n_features=1,
    noise=15,
    coef=True,
    random_state=42
)

# -----------------------------------------------------------
# Step 2: Train/test split
# -----------------------------------------------------------
# We reserve 20% of the data for final evaluation.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# -----------------------------------------------------------
# Step 3: Training the model
# -----------------------------------------------------------
# LinearRegression uses the normal equation by default.
modele = LinearRegression(fit_intercept=True)
modele.fit(X_train, y_train)

# Display learned coefficients
print(f"Coefficient (slope)     : {modele.coef_[0]:.4f}")
print(f"Intercept               : {modele.intercept_:.4f}")

# -----------------------------------------------------------
# Step 4: Evaluation on test data
# -----------------------------------------------------------
y_pred = modele.predict(X_test)

mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f"MSE  : {mse:.4f}")
print(f"R2   : {r2:.4f}")

# -----------------------------------------------------------
# Step 5: Visualization 1 - Scatter plot + regression line
# -----------------------------------------------------------
plt.style.use('seaborn-v0_8-whitegrid')
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Left chart: regression line
axes[0].scatter(X_train, y_train, alpha=0.4,
                color='steelblue',
                label="Training data", s=40)
axes[0].scatter(X_test, y_test, alpha=0.4,
                color='coral', label="Test data", s=40)

# Draw the regression line
x_line = np.linspace(X.min(), X.max(), 100).reshape(-1, 1)
y_line = modele.predict(x_line)
axes[0].plot(x_line, y_line, color='darkgreen',
             linewidth=2.5, label="Regression line")

axes[0].set_xlabel("Explanatory variable (x)", fontsize=12)
axes[0].set_ylabel("Target variable (y)", fontsize=12)
axes[0].set_title("Linear regression: line of best fit", fontsize=14)
axes[0].legend(fontsize=10)
axes[0].grid(True, alpha=0.3)

# -----------------------------------------------------------
# Step 6: Visualization 2 - Predictions vs Actual values
# -----------------------------------------------------------
axes[1].scatter(y_test, y_pred, alpha=0.6,
                color='purple', s=50, zorder=2)

# Ideal reference line (y = x)
min_val = min(y_test.min(), y_pred.min())
max_val = max(y_test.max(), y_pred.max())
axes[1].plot([min_val, max_val], [min_val, max_val],
             color='darkgreen', linewidth=2,
             linestyle='--',
             label='Perfect prediction')

axes[1].set_xlabel("Actual values", fontsize=12)
axes[1].set_ylabel("Predictions", fontsize=12)
axes[1].set_title("Predictions vs Actual values", fontsize=14)
axes[1].legend(fontsize=10)
axes[1].grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

This code covers the entire typical Machine Learning workflow: data generation, train/test split, training, evaluation, and visualization. The R² score (coefficient of determination) tells us the proportion of variance in the target variable explained by the model. An R² close to 1 indicates an excellent fit.

Hyperparameters

Here are the main hyperparameters of LinearRegression in scikit-learn:

Advantages

• Simplicity and interpretability: the coefficients $w_j$ have a direct interpretation. A positive coefficient means that an increase in the corresponding feature leads to an increase in the target.
• Training speed: thanks to the analytical solution, training is nearly instantaneous even on moderately sized datasets.
• No sensitive hyperparameters: there is almost nothing to tune, unlike neural networks or random forests.
• Solid baseline method: linear regression serves as an excellent baseline against which to compare more complex models. If a sophisticated model can’t beat linear regression, there’s a problem.
• Clear statistical assumptions: the Gauss-Markov theoretical framework guarantees that the estimators are the Best Linear Unbiased Estimators (BLUE) under well-defined conditions.

Limitations

• Linearity assumption: the model assumes a linear relationship between variables. If the actual relationship is nonlinear (quadratic, exponential, sinusoidal), predictions will be poor.
• Sensitivity to outliers: the MSE function heavily penalizes large errors, making the model very sensitive to unusual observations.
• Multicollinearity: when two explanatory variables are strongly correlated with each other, the coefficients become unstable and difficult to interpret. The matrix $X^\top X$ approaches singularity and the inverse is numerically unstable.
• Homoscedasticity assumption: the variance of errors is assumed to be constant. If it varies (heteroscedasticity), the confidence intervals are no longer reliable.
• High-dimensional limitations: when the number of features $p$ exceeds the number of samples $n$, the model overfits (fails to generalize).

Use Cases

1. Real Estate Forecasting

Estimate the price of an apartment based on its area, number of rooms, age of the building, and location. Linear regression provides a quick, interpretable first estimate of the price per square meter and the impact of each feature on the final price.

2. Sales Forecasting

Predict a company’s monthly revenue based on advertising budget, number of points of sale, seasonality, and economic indicators. The coefficients help identify which marketing lever has the most impact on sales.

3. Energy Consumption Estimation

Model the electricity consumption of a building from outdoor temperature, living area, thermal insulation, and number of occupants. Grid managers use these forecasts to balance energy supply and demand.

4. Dose Impact Analysis in Pharmacology

Understand how different doses of a drug influence a clinical measurement (e.g., reduction in blood pressure). Within a reasonable dosing range, the relationship is often approximately linear, making linear regression particularly well suited.

See Also

• Understanding the Exception to Zeckendorf’s Theorem in Python: Approach and Applications
• Digit Sum Correspondence: Mastering the Art of Coding in Python
• Become a « Trillionaire » Through Coding: How Python Can Revolutionize Your Financial Projects
• Mastering Colorful Charts in Python: Complete Guide for Beginners and Experts