Quadratic Discriminant Analysis: Complete Guide — Quadratic Discriminant Analysis

Summary — QDA (Quadratic Discriminant Analysis) is a probabilistic classifier that, like LDA, is based on Bayes’ theorem and the assumption of Gaussian class distributions. But unlike LDA which assumes a common covariance across all classes, QDA estimates a distinct covariance matrix for each class. This additional freedom allows modeling quadratic (elliptical, parabolic) decision boundaries rather than strictly linear ones. It is a compromise between the simplicity of LDA and the flexibility of non-parametric models.

Mathematical Principle

1. General Bayesian Framework

Like LDA, QDA is based on Bayes’ theorem for classification:

$$P(C_k | x) = \frac{P(x | C_k) \cdot P(C_k)}{P(x)}$$

where P(C_k|x) is the posterior probability that x belongs to class C_k, P(x|C_k) is the likelihood, and P(C_k) is the prior probability of class k.

The prediction is the class that maximizes this posterior probability:

$$\hat{y} = \arg\max_k P(C_k | x)$$

2. Per-class Gaussian Assumption

QDA assumes that the data of each class follow a Gaussian distribution with its own covariance:

$$P(x | C_k) = \mathcal{N}(x | \mu_k, \Sigma_k) = \frac{1}{(2\pi)^{p/2} |\Sigma_k|^{1/2}} \exp\left(-\frac{1}{2}(x – \mu_k)^T \Sigma_k^{-1} (x – \mu_k)\right)$$

This is where QDA fundamentally differs from LDA: each class k has its own covariance matrix Σ_k, whereas LDA imposes Σ_k = Σ for all classes.

3. Quadratic Discriminant Function

Taking the logarithm of the posterior and eliminating constant terms, we obtain the discriminant function for class k:

$$\delta_k(x) = -\frac{1}{2} \log|\Sigma_k| – \frac{1}{2}(x – \mu_k)^T \Sigma_k^{-1} (x – \mu_k) + \log(\pi_k)$$

where π_k = P(C_k) is the prior probability of class k.

The prediction is:

$$\hat{y} = \arg\max_k \delta_k(x)$$

4. Quadratic Decision Boundary

The boundary between two classes k and l is the set of points x where δ_k(x) = δ_l(x). Expanding this equality:

$$-\frac{1}{2}(x^T \Sigma_k^{-1} x – x^T \Sigma_l^{-1} x) + \text{linear terms} + \text{constants} = 0$$

The term x^T(Σ_k^{-1} – Σ_l^{-1})x is quadratic in x. This is what gives the algorithm its name: the decision boundary is a quadratic surface (ellipse, parabola, hyperbola).

5. Maximum Likelihood Parameter Estimation

Parameters are estimated on the training set:

• Mean of class k: $$\hat{\mu}k = \frac{1}{n_k} \sum x_i$$
• Covariance of class k: $$\hat{\Sigma}k = \frac{1}{n_k – 1} \sum_k)^T$$} (x_i – \hat{\mu}_k)(x_i – \hat{\mu
• Prior probability: $$\hat{\pi}_k = \frac{n_k}{n}$$

6. Key Difference: QDA vs LDA

Intuition

Imagine two groups of students on a campus: math students and art students.

LDA assumes that the spread (variability) of grades is identical in both groups. It will separate the groups with a straight line (in 2D) or a plane (in 3D). It’s like drawing an administrative border between two countries.

QDA says: “Wait, math grades are very concentrated (low variance), while art grades are very dispersed (high variance).” QDA then draws a curved boundary that encompasses the more spread-out group like an ellipse. As if the border between the two countries followed the natural shape of the terrain.

In summary: QDA allows each class to have its own “shape” in feature space. This is more flexible, but it requires more data to properly estimate each shape.

Python Implementation

Example 1: QDA from scratch

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

class QDAClassifier:
    """Quadratic Discriminant Analysis - implementation from scratch."""

    def __init__(self, reg_param=0.0):
        self.reg_param = reg_param  # Covariance regularization

    def fit(self, X, y):
        self.classes_ = np.unique(y)
        self.means_ = {}
        self.covs_ = {}
        self.priors_ = {}

        for k in self.classes_:
            X_k = X[y == k]
            self.means_[k] = np.mean(X_k, axis=0)
            self.covs_[k] = np.cov(X_k, rowvar=False)
            # Regularization for numerical stability
            if self.reg_param > 0:
                self.covs_[k] += self.reg_param * np.eye(self.covs_[k].shape[0])
            self.priors_[k] = len(X_k) / len(X)

        return self

    def _log_density(self, x, k):
        """Log of the Gaussian density for class k."""
        mu = self.means_[k]
        cov = self.covs_[k]
        prior = self.priors_[k]

        diff = x - mu
        cov_inv = np.linalg.inv(cov)
        _, log_det = np.linalg.slogdet(cov)

        # Quadratic discriminant function
        log_prior = np.log(prior)
        mahal = diff.T @ cov_inv @ diff
        log_det_term = -0.5 * log_det
        mahal_term = -0.5 * mahal

        return log_det_term + mahal_term + log_prior

    def predict(self, X):
        predictions = []
        for x in X:
            scores = {k: self._log_density(x, k) for k in self.classes_}
            predictions.append(max(scores, key=scores.get))
        return np.array(predictions)

# Data with non-linearly separable classes
X, y = make_classification(n_samples=600, n_features=2, n_informative=2,
    n_redundant=0, n_clusters_per_class=1, flip_y=0.1, random_state=42)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
    random_state=42)

# Our QDA
qda = QDAClassifier(reg_param=1e-4)
qda.fit(X_train, y_train)
qda_pred = qda.predict(X_test)
qda_acc = accuracy_score(y_test, qda_pred)
print(f"QDA accuracy: {qda_acc:.3f}")

# Display estimated parameters
for k in qda.classes_:
    print(f"Class {k}: mean={qda.means_[k].round(2)}, prior={qda.priors_[k]:.3f}")

Example 2: Visual LDA vs QDA comparison

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA

# Direct comparison
lda_model = LDA()
lda_model.fit(X_train, y_train)
lda_acc = accuracy_score(y_test, lda_model.predict(X_test))

qda_model = QDA(reg_param=1e-4)
qda_model.fit(X_train, y_train)
qda_acc2 = accuracy_score(y_test, qda_model.predict(X_test))

print(f"LDA accuracy: {lda_acc:.3f}")
print(f"QDA accuracy: {qda_acc2:.3f}")

# Visualizing decision boundaries
import matplotlib.pyplot as plt

xx, yy = np.meshgrid(np.linspace(X[:, 0].min()-1, X[:, 0].max()+1, 200),
                     np.linspace(X[:, 1].min()-1, X[:, 1].max()+1, 200))
grid = np.c_[xx.ravel(), yy.ravel()]

Z_lda = lda_model.predict(grid).reshape(xx.shape)
Z_qda = qda_model.predict(grid).reshape(xx.shape)

fig, axes = plt.subplots(1, 2, figsize=(14, 6))
for ax, Z, name, acc in zip(axes, [Z_lda, Z_qda], ['LDA', 'QDA'], [lda_acc, qda_acc2]):
    ax.contourf(xx, yy, Z, alpha=0.3, cmap='RdBu')
    ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap='RdBu',
        edgecolors='black', s=30)
    ax.set_title(f'{name} — Accuracy: {acc:.3f}')
    ax.set_xlabel('Feature 1')
    ax.set_ylabel('Feature 2')
plt.tight_layout()
plt.savefig('qda_vs_lda.png', dpi=150)

Example 3: QDA on data with different covariances

# Data specifically designed to favor QDA
np.random.seed(42)
# Class 0: low covariance
X0 = np.random.randn(200, 2) * 0.5 + np.array([2, 2])
y0 = np.zeros(200)
# Class 1: high covariance (elongated)
X1 = np.random.randn(200, 2) * np.array([2, 0.3]) + np.array([5, 2])
y1 = np.ones(200)

X_custom = np.vstack([X0, X1])
y_custom = np.concatenate([y0, y1])

X_tr, X_te, y_tr, y_te = train_test_split(X_custom, y_custom, test_size=0.3,
    random_state=42)

lda_c = LDA().fit(X_tr, y_tr)
qda_c = QDA().fit(X_tr, y_tr)

lda_c_acc = accuracy_score(y_te, lda_c.predict(X_te))
qda_c_acc = accuracy_score(y_te, qda_c.predict(X_te))

print(f"\nData with different covariances:")
print(f"LDA accuracy: {lda_c_acc:.3f}")
print(f"QDA accuracy: {qda_c_acc:.3f}")

Hyperparameters

Advantages of QDA

1. Non-linear boundaries: QDA captures quadratic relationships between classes that LDA completely misses. When classes have very different dispersions, QDA is significantly better.
2. Strong probabilistic foundation: QDA predictions are calibrated probabilities derived from Bayes’ theorem. Unlike SVM or KNN, which do not naturally produce probabilities.
3. No sensitive hyperparameters: Apart from covariance regularization, QDA has no parameters to tune like the C of an SVM or the k of a KNN. It is ready to use out of the box.
4. Interpretability: The estimated covariance matrices are directly interpretable: they describe how each class spreads in feature space.
5. Analytical solution: Like LDA, QDA has a closed-form maximum likelihood solution. No iterative optimization or grid search needed.

Limitations of QDA

1. Parameter explosion: For p features and K classes, QDA estimates K · p²/2 covariance parameters. With 100 features and 3 classes, that’s 15,000 parameters to estimate. Risk of overfitting on small datasets.
2. Need for lots of data: To properly estimate each covariance, you need at least p+1 examples per class, ideally many more. Below that the covariances are unstable.
3. Singularity of covariance matrices: If a class has fewer features than samples, or if a feature is collinear, the covariance becomes singular and non-invertible. Regularization is necessary.
4. Restrictive Gaussian assumption: If the classes do not follow a Gaussian distribution (e.g., multimodal, skewed), QDA performance drops.
5. Not suited for categorical data: QDA assumes continuous features. For mixed or categorical data, other approaches are needed.

4 Concrete Use Cases

1. Medical Diagnosis with Heterogeneous Profiles

In diagnosing cardiac pathologies, healthy patients and sick patients can present very different patterns: healthy patients have very homogeneous measurements (low variance), while sick patients show great variability depending on the type and stage of the disease. QDA naturally captures this asymmetry where LDA would fail.

2. Financial Fraud Detection

Legitimate transactions form a homogeneous, tightly clustered group in feature space (amount, time, location are consistent). Fraudulent transactions, by contrast, are very dispersed and unpredictable. QDA models this covariance asymmetry well and can detect frauds that LDA would miss by assuming the same dispersion for both categories.

3. Medical Image Classification

To distinguish healthy tissue from tumor tissue on MRI images, healthy tissue textures have stable characteristics (low variance), while tumors present very heterogeneous textures. QDA is particularly well-suited to this type of asymmetric classification.

4. Credit Scoring with Population Segments

In credit scoring, good borrowers have homogeneous financial profiles (stable income, steady employment), while defaulters can be so for very diverse reasons (job loss, illness, divorce, etc.). QDA models this profile asymmetry and improves accuracy compared to LDA.

Conclusion

QDA is the natural evolution of LDA when the equal-covariance assumption is unrealistic. By allowing each class to have its own shape in feature space, it captures non-linear relationships that LDA cannot model.

The trade-off is clear: more flexibility versus more parameters to estimate. On sufficiently large datasets with genuinely different covariances between classes, QDA is often superior. On small datasets or when covariances are similar, LDA remains the preferred choice.

In practice, testing LDA and QDA side by side with cross-validation is an excellent habit. If QDA significantly outperforms LDA, it is a strong signal that the classes have distinct variance structures that deserve to be modeled separately.

See Also

• Maîtriser les Diviseurs de Produits Binomiaux avec Python : Guide Complet
• Maîtriser les Composites à Propriété de Repunit Primaire avec Python : Guide et Applications