Logistic Regression: Complete Guide — Principles, Examples, and Python Implementation

Summary — Logistic regression is the classification algorithm par excellence. It models the probability of belonging to a class via a sigmoid function applied to a linear combination of features. Simple, interpretable, regularizable, and effective, it remains the essential starting point for any supervised classification problem, despite the rise of more complex methods.

Mathematical Principle

Logistic regression rests on four mathematical pillars:

1. Sigmoid function: The logistic (or sigmoid) function transforms a real value $z$ into a probability between 0 and 1:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Key properties: $\sigma(0) = 0.5$, $z \to +\infty \Rightarrow \sigma(z) \to 1$, $z \to -\infty \Rightarrow \sigma(z) \to 0$, and $\sigma’(z) = \sigma(z)(1-\sigma(z))$.

2. Probabilistic linear model: We set $z = w \cdot x + b$ (linear combination), then the probability of belonging to the positive class is:

$$P(y=1 | x) = \sigma(w \cdot x + b) = \frac{1}{1 + e^{-(w \cdot x + b)}}$$

The complementary probability is $P(y=0 | x) = 1 – P(y=1 | x)$.

3. Log-likelihood and cost function: The likelihood of the entire dataset is the product of individual probabilities:

$$L(w) = \prod_{i=1}^{n} p_i^{y_i} (1-p_i)^{1-y_i}$$

We maximize the log-likelihood (numerically more stable):

$$\ell(w) = \sum_{i=1}^{n} [y_i \log(p_i) + (1-y_i) \log(1-p_i)]$$

This is the negative equivalent of binary cross-entropy. The cost function to minimize is:

$$J(w) = -\frac{1}{n} \sum_{i=1}^{n} [y_i \log(p_i) + (1-y_i) \log(1-p_i)]$$

4. Gradient and optimization: The gradient has a remarkably simple form:

$$\frac{\partial \ell}{\partial w} = \sum_{i=1}^{n} (p_i – y_i) x_i$$

It has the same structure as linear regression, but with $p_i = \sigma(w \cdot x_i + b)$ instead of $w \cdot x_i + b$. Optimization is typically done via quasi-Newton methods (L-BFGS) or gradient descent.

5. Regularization:
– L2 (Ridge): adds $\frac{\lambda}{2} ||w||^2$ to the cost function — penalizes large weights, prevents overfitting.
– L1 (Lasso): adds $\lambda ||w||_1$ — forces some coefficients to zero, automatically performing variable selection.
– ElasticNet: linear combination of both: $\lambda_1 ||w||_1 + \frac{\lambda_2}{2} ||w||^2$.

Multinomial extension: For $K$ classes, the softmax function is used:

$$P(y=k | x) = \frac{e^{w_k \cdot x + b_k}}{\sum_{j=1}^{K} e^{w_j \cdot x + b_j}}$$

Intuition

Imagine a doctor who needs to diagnose a disease based on the patient’s temperature. A linear regression would say: “at 39°C, the predicted value is 0.84”. But what does 0.84 mean? It’s not a temperature, nor a clear diagnosis.

Logistic regression, on the other hand, says: “at 39°C, there is an 84% chance that the patient is sick”. This is a directly actionable answer. With a threshold of 50%, you classify as sick; by lowering the threshold to 30%, you become more cautious and hospitalize more patients as a precaution.

The voting analogy: Each feature contributes proportionally to its value and its weight. If “smoking” has a positive weight of +3, a smoker sees their probability of disease increase. If “exercising” has a negative weight of -1.5, an athlete sees their probability decrease. The final result is a synthetic probability that combines all these contributions.

Why is it so popular?: Logistic regression to machine learning is what a screwdriver is to a toolbox — simple, reliable, and surprisingly useful. Its greatest asset is interpretability: each coefficient quantifies exactly the impact of its feature on the target probability, which is crucial in regulated fields like finance or healthcare.

Python Implementation

Example 1: Binary Classification with scikit-learn

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix, roc_curve, roc_auc_score
import matplotlib.pyplot as plt
import numpy as np

# Dataset
X, y = make_classification(
    n_samples=1000, n_features=10, n_informative=5,
    n_redundant=3, n_classes=2, random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

# Model
model = LogisticRegression(
    penalty='l2',
    C=1.0,
    solver='lbfgs',
    max_iter=1000,
    random_state=42
)
model.fit(X_train, y_train)

# Predictions
y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)[:, 1]

# Evaluation
print(f"Accuracy: {model.score(X_test, y_test):.3f}")
print(f"AUC-ROC: {roc_auc_score(y_test, y_proba):.3f}")
print("\nClassification report:")
print(classification_report(y_test, y_pred))
print("\nConfusion matrix:")
print(confusion_matrix(y_test, y_pred))

Example 2: Interpreting Coefficients

import pandas as pd
import numpy as np

# Named features for interpretation
feature_names = [f"feature_{i}" for i in range(X.shape[1])]
coefficients = model.coef_[0]
intercept = model.intercept_[0]

# Coefficient table
coef_df = pd.DataFrame({
    'Feature': feature_names,
    'Coefficient': coefficients,
    'Odds Ratio (exp(β))': np.exp(coefficients),
    'Abs(Coeff)': np.abs(coefficients)
}).sort_values('Abs(Coeff)', ascending=False)

print("\nFeature importance (sorted by impact):")
print(coef_df[['Feature', 'Coefficient', 'Odds Ratio (exp(β))']].to_string(index=False))

# Visualization
plt.figure(figsize=(10, 5))
sorted_idx = np.argsort(coefficients)
colors = ['red' if c < 0 else 'green' for c in coefficients[sorted_idx]]
plt.barh(range(len(feature_names)), coefficients[sorted_idx], color=colors)
plt.yticks(range(len(feature_names)), [feature_names[i] for i in sorted_idx])
plt.axvline(x=0, color='black', linestyle='-', linewidth=0.5)
plt.title('Logistic Regression Coefficients')
plt.xlabel('Coefficient (positive = increases probability)')
plt.tight_layout()
plt.savefig('logistic_coefficients.png', dpi=150)
print("Coefficient chart saved")

Example 3: ROC Curve and Threshold Optimization

from sklearn.metrics import roc_curve, precision_recall_curve

# ROC curve
fpr, tpr, thresholds = roc_curve(y_test, y_proba)
roc_auc = roc_auc_score(y_test, y_proba)

plt.figure(figsize=(8, 5))
plt.plot(fpr, tpr, label=f'AUC = {roc_auc:.3f}', linewidth=2)
plt.plot([0, 1], [0, 1], 'k--', label='Random')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve — Logistic Regression')
plt.legend()
plt.grid(True, alpha=0.3)
plt.savefig('logistic_roc.png', dpi=150)

# Threshold optimization
precision, recall, pr_thresholds = precision_recall_curve(y_test, y_proba)
f1_scores = 2 * (precision * recall) / (precision + recall + 1e-8)
best_threshold = pr_thresholds[np.argmax(f1_scores)]
print(f"\nOptimized threshold (max F1): {best_threshold:.3f}")
print(f"F1 score at threshold 0.50: {f1_scores[np.argmin(np.abs(pr_thresholds - 0.5))]:.3f}")
print(f"F1 score at optimal threshold: {f1_scores.max():.3f}")

Example 4: Multinomial Classification

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

# Iris: 3 classes
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Multinomial classification
model_multi = LogisticRegression(
    penalty='l2',
    C=1.0,
    solver='lbfgs',
    multi_class='multinomial',  # softmax for K > 2 classes
    max_iter=1000,
    random_state=42
)
model_multi.fit(X_train, y_train)

print(f"Multiclass accuracy: {model_multi.score(X_test, y_test):.3f}")
print(f"Classes: {model_multi.classes_}")
print(f"Coefficient shape: {model_multi.coef_.shape}")

# Coefficient matrix (3 classes × 4 features)
for k, coef_k in enumerate(model_multi.coef_):
    print(f"Class {model_multi.classes_[k]}: coefficients = {coef_k.round(3)}")

Hyperparameters

Advantages of Logistic Regression

1. Maximum interpretability: Each coefficient is directly readable as a contribution to the probability via odds ratios. In regulated fields (banking, healthcare, insurance), this transparency is often a legal requirement.
2. Training speed: Log-likelihood optimization converges in seconds, even on millions of points. It is by far the fastest algorithm on the list for a classification problem.
3. Calibrated probabilities: Unlike SVM or KNN which require post-hoc recalibration (Platt scaling), logistic regression scores are already well-calibrated probabilities.
4. Built-in regularization: L1, L2, and ElasticNet are natively part of the model, allowing control of overfitting and automatic variable selection (with L1).
5. Uncontested baseline: It is the starting point for any classification problem. If a complex model can’t beat a simple logistic regression, the problem is probably poorly framed or the data doesn’t contain enough signal.

Limitations of Logistic Regression

1. Linearity: The decision boundary is inherently linear. If the classes are not linearly separable (XOR, concentric circles), the model will fail — unless you manually add non-linear features (interactions, polynomials).
2. Feature engineering required: To capture non-linear relationships, you must manually create interactions between features or polynomial transformations, which can be tedious and prone to overfitting if too many combinations are tested.
3. Sensitivity to multicollinearity: When two features are highly correlated, the coefficients become unstable and their signs can be counter-intuitive. L2 regularization mitigates this problem but does not solve it.
4. Log-odds linearity assumption: The model assumes that the logarithm of the odds is linear with respect to the features. If this assumption is violated, predictions will be biased, even with a lot of data.
5. Influential outliers: Extreme points can have a disproportionate impact on coefficients, especially without strong regularization. An outlier can “pull” the boundary toward it.

4 Concrete Use Cases

1. Banking Credit Scoring

Banks use logistic regression to assess the probability of a borrower defaulting. Each feature (income, number of payment incidents, age, loan duration) has a coefficient that reflects its weight in the risk of default. Odds ratios make it possible to explain to the customer why their application was denied — a European regulatory requirement (right to explanation).

2. Medical Diagnosis (Screening)

In public health, logistic regression models the probability that a patient will develop a disease based on their risk factors (BMI, smoking, heredity, age). The interpretability of the coefficients allows doctors to understand which factors weigh the most and to adapt recommendations. Additionally, calibrated probabilities allow setting screening thresholds adapted to the cost of false positives vs false negatives.

3. Churn Prediction in Telecom

Telecom operators use logistic regression to predict which customers are likely to cancel their subscription. Each coefficient identifies a churn factor (price too high, number of calls to customer service, seniority). This enables targeted actions: offering a discount to customers at high risk of leaving, rather than spending budget on all customers indiscriminately.

4. Marketing Campaign Results Analysis

In digital marketing, logistic regression evaluates the impact of each factor (acquisition channel, send time, demographic segment, promotional offer) on the probability of conversion. Odds ratios allow precise quantification: “Customers reached by email convert 2.3 times more than those reached by SMS, all other things being equal.” It is a powerful causal analysis tool when AB testing data is not available.

Best Practices

• Always normalize features before logistic regression with regularization (StandardScaler), otherwise L1/L2 penalties disproportionately affect features with large scales.
• Start with C=1.0 then vary from 0.01 to 100 via cross-validation to find the right level of regularization.
• Interpret odds ratios (exp(coef)) rather than raw coefficients — an odds ratio of 1.4 reads as “40% more likely per unit of feature”.
• Check probability calibration with a reliability diagram (CalibrationDisplay) — especially important if probabilities are used for decision-making.
• Handle class imbalance with class_weight='balanced' or resampling (SMOTE) if the positive class represents less than 10% of the data.

Conclusion

Logistic regression is arguably the most widely used algorithm in industry production, ahead of even Random Forests and neural networks. Its modesty is its strength: it doesn’t claim to solve everything, but offers an unmatched balance of performance, interpretability, and simplicity.

To choose between linear classifiers:
– Perceptron: when the data are linearly separable and you want an ultra-simple model.
– Linear SVM: when the maximum margin is important (SVM handles outliers near the boundary better).
– Logistic regression: in ALL other cases — it is the safest default choice.

See also

• Maîtriser la Logique Circulaire en Python : Guide Complet pour Débutants et Experts
• Maîtriser les Arrangements Maximaux en Python : Guide Complet et Astuces pour Développeurs