Multinomial Logistic Regression: Complete Guide — Principles, Examples, and Python Implementation

Summary: Multinomial logistic regression is the reference algorithm for supervised multi-class classification. It extends binary logistic regression through the softmax function, which produces a probability distribution over all classes. This guide covers the mathematical foundations, intuition, Python implementation with scikit-learn, hyperparameter tuning, and concrete use cases.

Mathematical principle

Softmax function

Multinomial logistic regression is based on the softmax function (sometimes called the normalized softmax function), which transforms a vector of raw scores into a probability distribution that sums to 1:

σ(z)k = exp(z_k) / Σ exp(z_j)}^{K

where:

• K is the total number of classes
• z_k is the linear score for class k, computed as z_k = w_kᵀx + b_k
• exp(·) is the exponential function
• σ(z)_k represents the estimated probability that observation x belongs to class k

Each class k has its own weight vector w_k and its own bias b_k, learned separately during training.

Prediction

The predicted class for an observation x is the one that maximizes the softmax probability:

ŷ = argmax_{k ∈ {1,…,K}} σ(z)_k

Cost function: categorical cross-entropy

Training multinomial logistic regression minimizes categorical cross-entropy, also called multi-class log-loss:

L(W, b) = -Σ{i=1}^{N} Σ{k=1}^{K} y_{ik} · log(σ(z^{(i)})_k)

where:

• N is the number of samples
• y_{ik} is 1 if sample i belongs to class k, 0 otherwise (one-hot encoding)
• σ(z^{(i)})_k is the softmax predicted probability for class k of sample i

This cost function heavily penalizes confident errors: if the model assigns a low probability to the true class, the penalty grows logarithmically.

Regularization

To avoid overfitting, an L1 (Lasso) or L2 (Ridge) regularization term is generally added to the cost function:

L_regularized = L(W, b) + λ · R(W)

• L2 regularization: R(W) = ||W||₂² (penalizes large weights)
• L1 regularization: R(W) = ||W||₁ (promotes sparsity, sets weights to zero)

The parameter C (inverse of λ) controls the regularization strength: a small C means strong regularization, and vice versa.

Intuition

From binary to multi-class classification

Binary logistic regression uses the sigmoid function to estimate P(y=1|x). But what do you do when there are more than two classes?

There are two main strategies:

One-vs-Rest (OvR): K independent binary classifiers are trained. Each classifier distinguishes one class from all the others. At prediction time, the class with the highest score is chosen. The problem? The probabilities are not normalized: their sum is not guaranteed to equal 1.

Multinomial (softmax): a single model is trained that directly produces a consistent probability distribution over the K classes. The probabilities always sum to 1, allowing direct probabilistic interpretation.

Why choose multinomial logistic regression?

Multinomial logistic regression is preferable to the OvR approach when:

1. Probabilities must be comparable between classes — softmax guarantees a consistent distribution.
2. Classes are mutually exclusive — each observation belongs to exactly one class.
3. Interpretability is important — weights can be analyzed as the influence of each feature on each class.

Unlike decision trees or random forests, this model remains linear in its decisions, making it fast, interpretable, and excellent as a baseline model.

Python Implementation

Environment and imports

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import load_iris, make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (
    accuracy_score, classification_report,
    confusion_matrix, ConfusionMatrixDisplay
)
import warnings
warnings.filterwarnings('ignore')

# Reproducibility
np.random.seed(42)

Loading data: the Iris dataset

The Iris dataset is a classic for illustrating multi-class classification. It contains 150 flower samples divided into 3 species, described by 4 morphological features.

# Load the Iris dataset
iris = load_iris()
X, y = iris.data, iris.target

# Display dataset information
print(f"Number of samples: {X.shape[0]}")
print(f"Number of features: {X.shape[1]}")
print(f"Number of classes: {len(np.unique(y))}")
print(f"Class names: {iris.target_names}")

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

# Standardization
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Model training

# Create and train the multinomial logistic regression model
model = LogisticRegression(
    multi_class='multinomial',   # Force the multinomial approach (softmax)
    solver='lbfgs',              # Default solver for multinomial
    max_iter=1000,               # Increase if non-convergence
    penalty='l2',                # L2 regularization
    C=1.0,                       # Inverse of regularization
    random_state=42
)

model.fit(X_train_scaled, y_train)

# Predictions
y_pred = model.predict(X_test_scaled)
y_proba = model.predict_proba(X_test_scaled)

# Evaluation
accuracy = accuracy_score(y_test, y_pred)
print(f"\nAccuracy: {accuracy:.4f}")
print(f"\nClassification report:")
print(classification_report(y_test, y_pred, target_names=iris.target_names))

Confusion matrix

# Confusion matrix
cm = confusion_matrix(y_test, y_pred)

plt.figure(figsize=(7, 5))
sns.heatmap(
    cm, annot=True, fmt='d', cmap='Blues',
    xticklabels=iris.target_names,
    yticklabels=iris.target_names
)
plt.ylabel('True label')
plt.xlabel('Prediction')
plt.title('Matrice de confusion — Régression logistique multinomiale (Iris)')
plt.tight_layout()
plt.show()

# Cross-validation
cv_scores = cross_val_score(
    LogisticRegression(
        multi_class='multinomial', solver='lbfgs',
        max_iter=1000, random_state=42
    ),
    scaler.fit_transform(X), y,
    cv=5, scoring='accuracy'
)
print(f"\nCross-validation (5 folds):")
print(f"  Mean: {cv_scores.mean():.4f} (+/- {cv_scores.std():.4f})")

Softmax probability visualization

# Visualize softmax probabilities for the first test samples
n_samples = 10
x_pos = np.arange(n_samples)
width = 0.25

fig, ax = plt.subplots(figsize=(10, 5))
for k in range(3):
    ax.bar(
        x_pos + k*width,
        y_proba[:n_samples, k],
        width,
        label=f'Class {iris.target_names[k]}'
    )

ax.set_xlabel('Sample')
ax.set_ylabel('Softmax probability')
ax.set_title('Distribution of softmax probabilities over 10 samples')
ax.set_xticks(x_pos + width)
ax.set_xticklabels([str(i) for i in range(n_samples)])
ax.legend()
ax.set_ylim(0, 1.1)
plt.tight_layout()
plt.show()

# Analyze model weights
fig, ax = plt.subplots(figsize=(10, 5))
feature_names = iris.feature_names
for k in range(3):
    ax.barh(
        [f'{n} → {iris.target_names[k]}' for n in feature_names],
        model.coef_[k],
        label=iris.target_names[k]
    )
ax.set_xlabel('Weight (coefficient)')
ax.set_title('Model weights by feature and class')
ax.legend()
plt.tight_layout()
plt.show()

Alternative example: synthetic 5-class data

To illustrate multinomial logistic regression on a more complex problem, let’s generate artificial data with 5 classes:

# Generate synthetic data
X_synth, y_synth = make_classification(
    n_samples=2000,
    n_features=10,
    n_informative=8,
    n_redundant=2,
    n_classes=5,
    n_clusters_per_class=1,
    random_state=42
)

X_s_train, X_s_test, y_s_train, y_s_test = train_test_split(
    X_synth, y_synth, test_size=0.3, random_state=42, stratify=y_synth
)

scaler_s = StandardScaler()
X_s_train_scaled = scaler_s.fit_transform(X_s_train)
X_s_test_scaled = scaler_s.transform(X_s_test)

# Training
model_s = LogisticRegression(
    multi_class='multinomial', solver='saga',
    max_iter=2000, penalty='l2', C=1.0, random_state=42
)
model_s.fit(X_s_train_scaled, y_s_train)

y_s_pred = model_s.predict(X_s_test_scaled)
print(f"Accuracy on synthetic data (5 classes): {accuracy_score(y_s_test, y_s_pred):.4f}")
print(f"\n{classification_report(y_s_test, y_s_pred)}")

Hyperparameters

The table below summarizes the essential hyperparameters of multinomial logistic regression in scikit-learn:

Quick solver selection guide

• lbfgs: robust and fast default solver, supports L2 and multinomial. Ideal as a first choice.
• saga: generalization of sag, supports L1, L2, elasticnet and multinomial. Recommended for large datasets.
• newton-cg: Newton’s method, fast on medium-sized data. Only supports L2.
• liblinear: does not support 'multinomial', only OvR. To be avoided for softmax.

Advantages and Limitations

Advantages

• Interpretability: coefficients are directly readable. A positive weight for class k indicates a positive association between the feature and that class.
• Calibrated probabilities: softmax produces well-calibrated probabilities, usable for decision-making and risk analysis.
• Speed: training is extremely fast, even on hundreds of thousands of samples.
• No sensitive hyperparameters: unlike random forests or neural networks, few adjustments are needed.
• Excellent baseline model: serves as a solid baseline. If a complex model does not beat multinomial logistic regression, it does not add value.
• Handles moderate collinearity well thanks to L2 regularization.

Limitations

• Linear decision boundaries: the model cannot capture complex non-linear relationships between features and classes.
• Sensitive to non-normalized features: standardization is almost mandatory for good results.
• Assumes independence of observations: like all generalized linear models.
• May underperform on highly non-linear data: SVMs with kernels, random forests, or neural networks will be superior in these cases.
• No native handling of missing values: imputation is required before training.
• Independence of Irrelevant Alternatives (IIA) assumption: multinomial logistic regression assumes that the ratio of probabilities between two classes does not depend on other classes, which may be unrealistic in some contexts.

Use cases

1. Image classification — Handwritten digit recognition (MNIST)

Multinomial logistic regression is a classic starting point for handwritten digit recognition. Each 28×28 image is flattened into a 784-pixel vector, and the model learns 10 weight vectors (one per digit from 0 to 9). Although a convolutional neural network surpasses this model in accuracy, multinomial logistic regression achieves about 92% accuracy in a few seconds, making it an essential baseline.

2. NLP — Multi-level sentiment classification

In natural language processing (NLP), text classification into several categories (e.g., very negative, negative, neutral, positive, very positive) commonly uses multinomial logistic regression. Input features are TF-IDF vectors or word embeddings. Each sentiment class gets its own decision hyperplane, and softmax distributes probabilities consistently. This approach is particularly effective when the volume of training data is limited.

3. Medical diagnostics — Multi-disease classification

In healthcare, a model can classify a patient’s symptoms among several possible pathologies (flu, COVID-19, cold, allergy, etc.). Multinomial logistic regression offers a crucial advantage here: softmax probabilities allow the physician to evaluate not only the most likely diagnosis, but also alternative diagnoses and their degree of likelihood. This transparency is essential in medicine, where false negatives can have serious consequences.

4. Customer segment prediction — Marketing and segmentation

In marketing, companies want to classify their customers into behavioral segments (loyal, opportunistic, dormant, at risk of leaving, new customer). Multinomial logistic regression uses features such as average basket, purchase frequency, recency of last interaction, or satisfaction score. Segment probabilities then allow personalizing campaigns: a customer with a 60% probability of being “at risk of leaving” will receive a targeted retention offer.

See also

• Implement the SHA-1 hashing algorithm in Python: Step-by-step guide
• Draw a Triangle with Circle Arcs in Python: Complete Guide for Developers