Quadratic Discriminant Analysis: Complete Guide — Principles, Examples, and Python Implementation

Summary — Quadratic Discriminant Analysis (QDA) is a probabilistic classification method that assigns each class its own covariance matrix, producing nonlinear decision boundaries. This guide covers the mathematical principles, geometric intuition, Python implementation with scikit-learn, and practical use cases.

Mathematical Principle

Quadratic Discriminant Analysis is based on Bayes’ theorem and the assumption that the data of each class follow a multivariate Gaussian distribution. Unlike Linear Discriminant Analysis (LDA), QDA does not assume that all classes share the same covariance matrix.

Discriminant Function

For each class k in {1, …, K}, we define the discriminant function:

δ_k(x) = -½ log|Σ_k| – ½ (x – μ_k)^ᵀ Σ_k⁻¹ (x – μ_k) + log(π_k)

Where:

• x ∈ Rᵖ is the vector of observations to classify (p features),
• μ_k is the mean vector of class k,
• Σ_k is the class-specific covariance matrix for class k,
• π_k is the prior probability of belonging to class k,
• |Σ_k| denotes the determinant of the covariance matrix.

Classification Rule

The observation x is assigned to the class that maximizes the discriminant function:

ŷ = argmax_k δ_k(x)

Parameter Estimation

Parameters are estimated by maximum likelihood from the training data:

• Class mean: μ̂k = (1 / n_k) Σ x_i
• Class covariance: Σ̂k = (1 / n_k) Σ (x_i – μ̂_k)(x_i – μ̂_k)^ᵀ
• Prior probability: π̂_k = n_k / n

where n_k is the number of observations in class k and n is the total number of observations.

Decision Boundaries

The boundary between two classes k and l is the set of points satisfying δ_k(x) = δ_l(x). Since each class has its own covariance matrix, this equation involves quadratic terms in x, giving quadratic boundaries (ellipses, parabolas, or hyperboles depending on the geometry of the covariances).

Conversely, in the case of LDA where Σ_k = Σ for all k, the quadratic terms cancel out and the boundaries are linear (hyperplanes in the feature space).

Geometric Intuition

Imagine that each class forms a point cloud in the feature space. In QDA, each cloud has its own shape:

• LDA: all clouds have the same overall shape (same covariance ellipse), only their center position differs. The boundaries are straight lines (or hyperplanes in higher dimensions).
• QDA: each cloud has its own covariance ellipse — some classes may be very stretched in one direction, others more compact and spherical. The boundaries that separate the classes are quadratic curves: parabolic arcs, hyperbolic branches, or elliptical arcs.

Why Quadratic Curves?

Consider a simple example in two dimensions with two classes:

• Class A: a very horizontally elongated cloud (high variance in x, low in y).
• Class B: a round, compact cloud (similar variances in all directions).

Where the two clouds approach each other, the boundary separating them will not be a straight line: it will follow the natural curvature of the covariance ellipses. This flexibility allows QDA to capture situations where classes have heterogeneous dispersions.

The Bias-Variance Tradeoff

This power comes at a cost. In LDA, a single shared covariance matrix is estimated (p(p+1)/2 parameters for p variables). In QDA, one is estimated per class (K × p(p+1)/2 parameters). For p = 50 and K = 5, this represents:

• LDA: 1,275 covariance parameters.
• QDA: 6,375 covariance parameters.

With little training data, QDA risks overfitting because it estimates too many parameters relative to the amount of information available.

Python Implementation

Here is a complete implementation of Quadratic Discriminant Analysis with scikit-learn, including a visual comparison between LDA and QDA.

Installing Dependencies

pip install scikit-learn numpy matplotlib seaborn

Basic Example

import numpy as np
import matplotlib.pyplot as plt
from sklearn.discriminant_analysis import (
    LinearDiscriminantAnalysis,
    QuadraticDiscriminantAnalysis,
)
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, accuracy_score

# Generate data with classes of different dispersions
rng = np.random.RandomState(42)

# Class 0: centered at (-2, -2), small isotropic covariance
n0 = 300
X0 = rng.multivariate_normal(
    mean=[-2, -2],
    cov=[[0.5, 0], [0, 0.5]],
    size=n0
)
y0 = np.zeros(n0, dtype=int)

# Class 1: centered at (2, 2), large elliptical covariance
n1 = 300
X1 = rng.multivariate_normal(
    mean=[2, 2],
    cov=[[3, 1.5], [1.5, 2]],
    size=n1
)
y1 = np.ones(n1, dtype=int)

# Merge data
X = np.vstack([X0, X1])
y = np.concatenate([y0, y1])

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

# Train LDA and QDA
lda = LinearDiscriminantAnalysis()
qda = QuadraticDiscriminantAnalysis()

lda.fit(X_train, y_train)
qda.fit(X_train, y_train)

# Predictions and evaluation
for name, model in [("LDA", lda), ("QDA", qda)]:
    y_pred = model.predict(X_test)
    acc = accuracy_score(y_test, y_pred)
    print(f"--- {name} ---")
    print(f"Accuracy: {acc:.4f}")
    print(classification_report(y_test, y_pred, target_names=["Class 0", "Class 1"]))

Visualizing Decision Boundaries

def plot_decision_boundary(X, y, models, names):
    fig, axes = plt.subplots(1, len(models), figsize=(6 * len(models), 5))

    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = np.meshgrid(
        np.linspace(x_min, x_max, 300),
        np.linspace(y_min, y_max, 300)
    )
    grid = np.c_[xx.ravel(), yy.ravel()]

    for ax, model, name in zip(axes, models, names):
        Z = model.predict(grid).reshape(xx.shape)
        probs = model.predict_proba(grid)[:, 1].reshape(xx.shape)

        ax.contourf(xx, yy, Z, alpha=0.3, cmap=plt.cm.RdYlBu, levels=[-0.5, 0.5, 1.5])
        ax.contour(xx, yy, probs, levels=[0.5], colors="black", linewidths=2)

        ax.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.RdYlBu,
                   edgecolors="black", s=40, alpha=0.8)

        acc = accuracy_score(y, model.predict(X))
        ax.set_title(f'{name}\nAccuracy: {acc:.3f}', fontsize=14)
        ax.set_xlabel("x₁")
        ax.set_ylabel("x₂")

    plt.tight_layout()
    plt.show()

# Apply
plot_decision_boundary(X, y, [lda, qda], ["LDA", "QDA"])

Comparison with the Optimal Bayesian Classifier

QDA is a direct application of the optimal Bayesian classifier under the Gaussian assumption per class. If the data actually follow this assumption, QDA converges to the optimal classifier as the sample size tends to infinity.

print("Covariance matrices estimated by QDA:")
for k, cov in enumerate(qda.covariance_):
    print(f"Class {k}:")
    print(cov)
    print(f"  Determinant: {np.linalg.det(cov):.4f}")

Hyperparameters

The QuadraticDiscriminantAnalysis object in scikit-learn has three main hyperparameters:

Choosing Regularization

When the number of features p is large relative to the number of observations per class n_k, the estimated covariance matrices can become singular (non-invertible). The reg_param parameter adds λI to each Σ̂_k:

Σ̃_k = Σ̂_k + λI

This Tikhonov-type regularization ensures the matrices remain invertible and stabilizes predictions.

from sklearn.model_selection import GridSearchCV

param_grid = {'reg_param': [0.0, 0.001, 0.01, 0.05, 0.1, 0.5]}
qda_grid = QuadraticDiscriminantAnalysis(store_covariance=True)
grid_search = GridSearchCV(qda_grid, param_grid, cv=5, scoring="accuracy")
grid_search.fit(X_train, y_train)

print(f"Best reg_param: {grid_search.best_params_['reg_param']}")
print(f"Best accuracy (cross-validation): {grid_search.best_score_:.4f}")

Advantages and Limitations

Advantages

• Boundary flexibility: QDA captures nonlinear separations between classes, where LDA would fail.
• Heterogeneous covariances: Ideal when classes have markedly different dispersions (this is precisely the main use case).
• Probabilistic classification: Provides membership probabilities via predict_proba(), useful for thresholding and risk analysis.
• No complex hyperparameters: Unlike random forests or neural networks, QDA has virtually no hyperparameters to tune.
• Interpretability: The estimated parameters (means, covariances, prior probabilities) have a direct statistical meaning.

Limitations

• Number of parameters: K × p(p+1)/2 covariance parameters to estimate. This can become prohibitive with many features.
• Overfitting risk: With little training data and many features, the model memorizes noise instead of learning the real structure.
• Gaussian assumption: If class distributions strongly deviate from multivariate normality, performance may degrade.
• Singular matrices: When n_k ≤ p for a class, the estimated covariance matrix is singular — regularization (reg_param > 0) is required.
• Prediction complexity: O(K × p²) per prediction, compared to O(K × p) for LDA. Noticeable impact in real time with high dimensions.

Use Cases

1. Medical Diagnosis — Biopsy Analysis

In oncology, healthy and tumor tissues often present molecular signatures with radically different variances: cancer cells show much greater expression heterogeneity than healthy cells. QDA, by allowing distinct covariances, naturally captures this asymmetry. Studies on breast cancer classification from transcriptomic data have shown that QDA outperformed LDA by 4 to 7 accuracy points thanks to this capability.

2. Finance — Banking Fraud Detection

Fraudulent transactions form a class whose distribution is fundamentally different from legitimate transactions: unusual amounts, atypical hours, improbable locations. The dispersion of fraudulent transactions is inherently greater and of a different structure. QDA models these two regimens separately, producing curved decision boundaries that better fit the real geometry of the data. The analysis protocol requires precise threshold calibration, as a false negative can cost thousands of euros.

3. Speech Recognition

In speech recognition, each phoneme is modeled by a Gaussian distribution in the cepstral coefficient (MFCC) space. Phonemes do not have the same acoustic variability: some are very stable, others vary enormously depending on the speaker. QDA allows each phoneme to be assigned its own covariance model, improving classification accuracy compared to LDA, especially for high-variability phonemes. This approach also works well for speaker accent recognition.

4. Remote Sensing — Satellite Image Classification

Each type of land cover — forest, urban area, coastal waters, agricultural crops — has a spectral signature with a characteristic dispersion. Urban areas present much stronger spectral heterogeneity than water bodies, which reflect light uniformly. QDA exploits this difference in covariance structure to produce more accurate classifiers, especially at class boundaries where distributions overlap asymmetrically. This approach is particularly useful for high-resolution land-use mapping.

QDA vs LDA: When to Choose Which?

Practical rule: If the classes in your problem have visibly different dispersions (verifiable by exploratory analysis of per-class variances), start with QDA. Otherwise, LDA is a more robust starting point.

See Also

• Implementation of Cramer’s Algorithm and Determinant Calculation in Python
• Optimize Your Python Code with SOP and POS: Complete Guide for Beginners