PCA (Principal Component Analysis): Complete Guide — Principles, Examples, and Python Implementation

Summary

Principal Component Analysis (PCA), or Analyse en Composantes Principales (ACP) in French, is the most fundamental and widely used unsupervised dimensionality reduction algorithm in data science. It transforms a set of potentially correlated variables into a new set of linearly independent variables, called principal components, ordered by the amount of variance they capture. The result: fewer dimensions, less noise, and data that remains interpretable.

In this complete guide, we explore the mathematical foundations of PCA, its geometric intuition, its practical implementation with scikit-learn, its hyperparameters, as well as its strengths and limitations.

Mathematical Principle of PCA

PCA relies on a logical sequence of algebraic operations. Understanding each step is essential to apply it correctly and interpret its results.

1. Centering and Scaling

Let a data matrix X of size n × p, where n is the number of observations and p is the number of variables. The first step consists of centering and scaling each variable:

For each variable j (column j of X):

• Centering: subtract the mean of the variable from each observation.
  x’_ij = x_ij − x̄_j
• Scaling: divide by the standard deviation of the variable.
  x’‘_ij = (x_ij − x̄_j) / σ_j

The resulting matrix Z is centered-scaled: each column has zero mean and a standard deviation equal to 1. This step is crucial because variables measured in different units (for example, kilograms and centimeters) would otherwise have incomparable variance scales, and PCA would be biased toward large-scale variables.

2. Covariance Matrix

We compute the covariance matrix of the centered-scaled matrix Z:

C = (1/n) × Zᵀ × Z

This matrix C is of size p × p. It is:

• Symmetric: C_ij = C_ji
• Positive definite (or positive semi-definite): all its eigenvalues are positive or zero
• Its diagonal entries equal 1 (because the variables have been scaled)
• Its off-diagonal entries represent the correlation coefficients between variables

When the data is not scaled but only centered, the diagonal entries represent the variances of each variable, and the off-diagonal entries the covariances.

3. Eigenvalue and Eigenvector Decomposition

The core of PCA is the spectral decomposition of the covariance matrix C. We seek the eigenvalues λ_i and the eigenvectors v_i such that:

C × v_i = λ_i × v_i

For a p × p matrix, we obtain p pairs (λ_i, v_i) ordered by decreasing eigenvalues:

λ₁ ≥ λ₂ ≥ … ≥ λ_p ≥ 0

The key properties are:

• Each eigenvalue λ_i represents the amount of variance captured by the i-th principal component.
• Each eigenvector v_i defines the direction of that principal component in the original p-dimensional space.
• The eigenvectors are orthogonal to each other: they form an orthonormal basis of the space.
• The sum of the eigenvalues equals the trace of the covariance matrix, i.e., the total variance: Σλ_i = trace(C) = p (for scaled data).

4. Projection onto the First k Components

We choose to keep only the first k principal components (with k < p). The projection matrix V_k is formed from the first k eigenvectors (columns). The transformation is written:

T = Z × V_k

where T is the transformed matrix of size n × k. Each observation is now represented in a k-dimensional space instead of p.

5. Explained Variance

The proportion of variance explained by the i-th principal component is calculated as:

Variance explained (component i) = λ_i / Σλ_j

The cumulative explained variance of the first k components is:

Cumulative variance = (Σ_{i=1}^{k} λ_i) / Σλ_j

A common choice is to select k such that the cumulative variance exceeds a threshold, typically 90% or 95%. This ensures that most of the information is preserved despite the dimensionality reduction.

Geometric Intuition

Imagine you are observing a three-dimensional object — say, an elongated rugby ball. If you look at it head-on, you only see a flat circle. If you look from the side, you perceive its true elongated shape. PCA does exactly this with your data: it automatically finds the “best angle” to look at it.

More precisely:

1. Find the direction of greatest dispersion: PCA identifies the line along which the data points are most spread out. This is the first principal component. If your data forms an elliptical cloud, it is the major axis of the ellipse.
2. Find the orthogonal direction of second greatest dispersion: The second principal component is perpendicular to the first and captures the maximum remaining variance. It is the minor axis of the ellipse.
3. Project the data onto these axes: instead of describing each point by its p original coordinates, we describe it by its coordinates along these new directions. We have lost some information (the variance of the discarded dimensions), but we have kept the essential part.

This geometric interpretation explains why PCA is so effective: if your data naturally lives in a low-dimensional subspace (which is common), PCA discovers it automatically.

Python Implementation with scikit-learn

Let’s see how to implement PCA in practice with the scikit-learn library.

Basic Example with the Iris Dataset

from sklearn.decomposition import PCA
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
import numpy as np

# Load data
iris = load_iris()
X, y = iris.data, iris.target

# Step 1: Centering and scaling
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Step 2: Full PCA (4 components)
pca = PCA(n_components=4)
X_pca = pca.fit_transform(X_scaled)

# Analyze results
print("Eigenvalues (variances):", pca.explained_variance_)
print("Explained variances:", pca.explained_variance_ratio_)
print("Cumulative explained variance:", np.cumsum(pca.explained_variance_ratio_))
print("Components (eigenvectors):\n", pca.components_)

The typical result shows that the first two components explain more than 95% of the total variance:

Eigenvalues (variances): [2.9108 0.9212 0.1474 0.0206]
Explained variances: [0.7296 0.2303 0.0368 0.0032]
Cumulative explained variance: [0.7296 0.9599 0.9967 1.0000]

Scree Plot

The scree plot visualizes the variance explained by each component and helps choose k:

plt.figure(figsize=(8, 5))
components = range(1, len(pca.explained_variance_ratio_) + 1)

plt.bar(components, pca.explained_variance_ratio_,
        alpha=0.7, label='Explained variance', color='steelblue')
plt.plot(components, np.cumsum(pca.explained_variance_ratio_),
         'ro-', label='Cumulative variance', linewidth=2)

plt.axhline(y=0.95, color='green', linestyle='--', label='95% threshold')
plt.xlabel('Principal component number')
plt.ylabel('Proportion of explained variance')
plt.title('Scree Plot')
plt.legend()
plt.grid(True, alpha=0.3)
plt.xticks(components)
plt.tight_layout()
plt.show()

2D Visualization After Reduction

# Reduction to 2 dimensions
pca_2d = PCA(n_components=2)
X_2d = pca_2d.fit_transform(X_scaled)

plt.figure(figsize=(8, 6))
for target, color in zip([0, 1, 2], ['red', 'green', 'blue']):
    mask = y == target
    plt.scatter(X_2d[mask, 0], X_2d[mask, 1],
                c=color, label=iris.target_names[target],
                alpha=0.7, edgecolors='black', s=80)

plt.xlabel(f'Component 1 ({pca_2d.explained_variance_ratio_[0]*100:.1f}% variance)')
plt.ylabel(f'Component 2 ({pca_2d.explained_variance_ratio_[1]*100:.1f}% variance)')
plt.title('2D PCA Projection — Iris Dataset')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

Biplot (Combining Observations and Variables)

The biplot overlays the projected observations and the contributions of the original variables:

plt.figure(figsize=(10, 7))

# Projected points
for target, color in zip([0, 1, 2], ['red', 'green', 'blue']):
    mask = y == target
    plt.scatter(X_2d[mask, 0], X_2d[mask, 1],
                c=color, label=iris.target_names[target], alpha=0.6, s=60)

# Original variable arrows
for i, feature in enumerate(iris.feature_names):
    # Scaled eigenvector coordinates
    x_arrow = pca_2d.components_[0, i] * np.sqrt(pca_2d.explained_variance_[0])
    y_arrow = pca_2d.components_[1, i] * np.sqrt(pca_2d.explained_variance_[1])
    plt.arrow(0, 0, x_arrow * 3, y_arrow * 3,
              color='black', head_width=0.15, head_length=0.1)
    plt.text(x_arrow * 3.2, y_arrow * 3.2, feature, fontsize=11, fontweight='bold')

plt.axhline(0, color='gray', linewidth=0.5)
plt.axvline(0, color='gray', linewidth=0.5)
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')
plt.title('PCA Biplot — Iris')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

Automatic PCA by Variance Threshold

# Keep enough components to explain 95% of the variance
pca_auto = PCA(n_components=0.95)
X_reduced = pca_auto.fit_transform(X_scaled)

print(f"Original dimensions: {X.shape[1]}")
print(f"Dimensions after PCA: {X_reduced.shape[1]}")
print(f"Variance retained: {sum(pca_auto.explained_variance_ratio_)*100:.1f}%")

PCA Hyperparameters in scikit-learn

n_components

This is the most important parameter. It controls the number of components to keep:

• Integer (e.g., n_components=3): keeps exactly that many components.
• Float between 0 and 1 (e.g., n_components=0.95): automatically keeps the minimum number of components needed to explain at least that proportion of variance.
• ‘mle’: uses Minka’s maximum likelihood estimation to automatically determine the optimal number of components.
• None (default): keeps all components.

svd_solver

The singular value decomposition solver:

• ‘full’: full SVD via LAPACK. Accurate but slow for very large matrices.
• ‘auto’ (default): automatically chooses based on data size.
• ‘arpack’: uses ARPACK. Efficient when k is small relative to p.
• ‘randomized’: randomized SVD. Very fast for large matrices when only a few components are needed.

whiten

When whiten=True, each component is divided by the square root of its eigenvalue, which normalizes the variance of all components to 1. This can improve the performance of certain downstream algorithms (such as SVMs), but removes information about the relative importance of components.

tol

Tolerance for the ‘arpack’ solver. Default 0.0. Controls convergence precision.

iterated_power

Number of iterations for the ‘randomized’ solver. Default ‘auto’ (usually sufficient). Increasing this parameter improves precision at the cost of computation time.

Advantages and Limitations of PCA

Advantages

• Extremely fast: singular value decomposition is highly optimized and runs in milliseconds for reasonably sized data.
• Interpretable: the components are linear combinations of the original variables. You can examine the eigenvector coefficients to understand what each component represents.
• No sensitive hyperparameter: the choice of the number of components is guided by explained variance, an objective criterion.
• Universal: works with any type of numerical data; no distributional assumption is strictly required for application (although statistical interpretation is optimal with Gaussian data).
• Denoises data: by discarding low-variance components, measurement noise is often eliminated.

Limitations

• Linearity: PCA only captures linear relationships. If the underlying structure of your data is nonlinear (data varies along a curve or curved surface), PCA will give poor results. This is where methods like t-SNE or UMAP take over.
• Sensitivity to outliers: extreme values artificially inflate variance and can distort the principal components. Robust preprocessing (median, IQR) is recommended in the presence of outliers.
• Loss of semantic interpretability: each component is a mixture of all the original variables. If the variables had a clear meaning (price, height, weight), the components may lose it.
• Unsuitable for categorical data: PCA requires continuous numerical data. For categorical data, Multiple Correspondence Analysis (MCA) is preferred.

4 Concrete Use Cases

1. Image Compression

Images can be seen as matrices of pixels. PCA allows compressing an image by keeping only the first k principal components. With only 50 components on a 1,000-pixel-wide image, one often obtains a reconstruction visually indistinguishable from the original, while reducing data size by 95%.

2. Preprocessing for Classification

Before training a classifier (logistic regression, random forest, SVM) on high-dimensional data, PCA reduces the number of variables, which speeds up training and reduces the risk of overfitting. This is a classic step in machine learning pipelines for genomic data (thousands of genes) or text data (high-dimensional bags of words).

3. High-Dimensional Data Visualization

Projecting data from 50, 100, or 1,000 dimensions into 2D or 3D for visualization is one of the most frequent uses of PCA. Although t-SNE and UMAP often give more separated visualizations for complex data, PCA remains the reference for its speed and reproducibility.

4. Anomaly Detection

Anomalous points typically have high reconstruction scores: when projected onto the principal components and then reconstructed in the original space, the reconstruction error is large. By computing this error for each observation, outliers can be identified — a technique used in fraud detection, industrial monitoring, and cybersecurity.

Conclusion

PCA is a fundamental tool in every machine learning practitioner’s toolkit. Its mathematical simplicity, execution speed, and interpretability make it often the first method to try for any dimensionality reduction task. Even though more sophisticated methods exist for specific cases (t-SNE, UMAP, autoencoders), PCA remains unmatched as a starting point — fast, reliable, and always informative.

See Also

• Understanding Idempotent Matrices in Python: Complete Guide for Developers
• Mastering Crossed Grid Manipulation with Python: Ultimate Guide to Managing Rectangles