Principal Component Analysis

Principal Component Analysis (PCA) is an unsupervised dimensionality reduction technique that finds the directions of greatest variance in a dataset and projects the data onto a new, smaller set of uncorrelated axes called principal components. It matters in modern AI because high-dimensional data (images, text embeddings, genomic data) is computationally expensive and often noisy — PCA strips away redundancy and keeps only what varies most. The counterintuitive fact: PCA does not select existing features; it invents entirely new ones that are linear combinations of all the originals, meaning the resulting components rarely have a simple human-readable interpretation.

The core insight

Your data almost never fills the high-dimensional space it lives in — most of the action is concentrated along a few directions. PCA finds those directions automatically by rotating your coordinate system so the first axis points where the data spreads out the most, the second axis points where it spreads out the second most (at a right angle to the first), and so on. Once you have those axes, you can safely throw away the ones with tiny spread — the data loses very little real information, but becomes much cheaper to store and process. The "aha moment" is realising that throwing away dimensions does not mean throwing away the same old columns; it means discarding directions in which your data is almost flat.

How It Works

PCA works in a small number of concrete steps. Understanding each step, even at a high level, makes the algorithm feel completely natural rather than magical.

Step 1 — Center the data. Subtract the mean of each feature so the data cloud sits at the origin.

\quad \tilde{X} = X - \mu

Explanation: Shifting the data to be centered ensures PCA measures spread around the middle of the cloud, not around zero. Here $X$ is the original data matrix (rows = samples, columns = features), $\mu$ is the row vector of per-feature means, and $\tilde{X}$ is the centered result.

Step 2 — Compute the covariance matrix. This square matrix captures how much every pair of features varies together.

\quad C = \frac{1}{n-1}\,\tilde{X}^{\top}\tilde{X}

Explanation: Each entry $C_{ij}$ measures whether features $i$ and $j$ tend to rise and fall together — large positive values mean they are correlated. $n$ is the number of data samples, and $\top$ denotes the matrix transpose.

Step 3 — Find the eigenvectors and eigenvalues. The eigenvectors of $C$ are the principal component directions; the eigenvalues tell you how much variance each direction captures.

\quad C\,\mathbf{v} = \lambda\,\mathbf{v}

Explanation: This equation says: a principal component direction $\mathbf{v}$ is special because the covariance matrix only stretches it (by factor $\lambda$ ) without rotating it. $\mathbf{v}$ is an eigenvector (one principal component axis), and $\lambda$ is its eigenvalue (the amount of variance explained).

Step 4 — Project the data. Multiply the centered data by the chosen eigenvectors to get the new, lower-dimensional representation.

\quad Z = \tilde{X}\,V_k

Explanation: Each row of $Z$ is a sample expressed in the new coordinate system made of the top $k$ principal components. $V_k$ is the matrix whose columns are the $k$ eigenvectors with the largest eigenvalues, and $Z$ is the transformed dataset. --- Here is a complete, minimal Python example using scikit-learn: ```python import numpy as np import matplotlib.pyplot as plt from sklearn.decomposition import PCA from sklearn.datasets import load_iris from sklearn.preprocessing import StandardScaler

# 1. Load data — Iris has 4 features, 150 samples X, y = load_iris(return_X_y=True)

# 2. Standardise: mean=0, std=1 for each feature

# (important when features have different units/scales) X_scaled = StandardScaler().fit_transform(X)

# 3. Fit PCA, keep the top 2 principal components pca = PCA(n_components=2) X_pca = pca.fit_transform(X_scaled) print("Original shape :", X_scaled.shape) # (150, 4) print("Reduced shape :", X_pca.shape) # (150, 2)

# 4. How much variance do the 2 components explain? explained = pca.explained_variance_ratio_ print(f"PC1 explains {explained[0]:.1%} of variance") print(f"PC2 explains {explained[1]:.1%} of variance") print(f"Total retained: {sum(explained):.1%}")

# 5. Plot the reduced data, colour-coded by species colours = ['#60a5fa', '#d4a853', '#e2e8f0'] for cls in range(3): mask = y == cls plt.scatter(X_pca[mask, 0], X_pca[mask, 1], c=colours[cls], label=f'Class {cls}', alpha=0.8) plt.xlabel('Principal Component 1') plt.ylabel('Principal Component 2') plt.title('Iris dataset — PCA projection') plt.legend() plt.tight_layout() plt.show() ``` Running this reveals that just 2 components already capture roughly 95 % of all variance in the 4-feature Iris dataset — an enormous compression with almost no information loss. ```python

# Choosing how many components to keep: the "elbow" method pca_full = PCA() # keep all components pca_full.fit(X_scaled) cumulative = np.cumsum(pca_full.explained_variance_ratio_) plt.plot(range(1, len(cumulative)+1), cumulative, marker='o', color='#60a5fa') plt.axhline(0.95, color='#d4a853', linestyle='--', label='95% threshold') plt.xlabel('Number of components') plt.ylabel('Cumulative explained variance') plt.title('Scree / Elbow plot') plt.legend() plt.tight_layout() plt.show() ``` The elbow plot is the practitioner's standard tool: pick the number of components where the curve bends — adding more components after that point gives diminishing returns.

researcher · Karl Pearson | 1857 – 1936 | Inventor of PCA

Pearson was a British mathematician and statistician who founded the world's first university statistics department at University College London. In his landmark 1901 paper "On Lines and Planes of Closest Fit to Systems of Points in Space," he introduced the core idea of PCA: finding the axis that minimises perpendicular distances from data points, which is mathematically equivalent to finding the direction of maximum variance. Surprisingly, Pearson was also a committed eugenicist — a reminder that powerful statistical tools can originate with deeply flawed individuals, and that the tools must be judged separately from the person.

researcher · Harold Hotelling | 1895 – 1973 | Modern Formulation

Hotelling was an American mathematical statistician and economist who independently rediscovered and greatly extended Pearson's idea in his 1933 paper "Analysis of a Complex of Statistical Variables into Principal Components." He gave PCA its modern name, framed it in terms of the eigendecomposition of the covariance matrix (the formulation still used today), and showed how to compute successive components that are orthogonal to one another. He is also famous in economics for Hotelling's Law, which explains why competing stores cluster together — a sign of the unusually broad reach of his thinking.

Figure

PCA Step-by-Step Data Flow

Figure

2D PCA Rotation Intuition

Intuition

The Mental Model

Imagine you are photographing a crumpled piece of paper from directly above — the photo is a 2D projection of a 3D object. If the paper lies almost flat, the top-down photo captures nearly all the interesting wrinkles and shapes; you lose almost nothing by going from 3D to 2D. PCA is the algorithm that automatically finds that best "camera angle" for your data. A second analogy: think of a shadow puppet show. The puppet (your data) exists in 3D, but the shadow on the wall (the projection) is 2D. PCA finds the light source angle that produces the most informative, least squished shadow. A shadow from the wrong angle turns a complex puppet into a featureless blob — PCA avoids that by always choosing the angle that preserves the most detail. More concretely: if you surveyed people measuring both their shoe size and their height, those two numbers are highly correlated — tall people tend to have big feet. PCA would find a single new axis running diagonally through the (height, shoe-size) scatter plot, roughly meaning "overall body size." That one axis captures almost all the variation; the perpendicular axis (roughly "unusually big feet for your height") explains almost nothing. You could safely keep only the first axis and lose almost no information.

Historical Development

Origins · 1 minhow this was discovered

History

1901

Karl Pearson publishes "On Lines and Planes of Closest Fit to Systems of Points in Space" — the first mathematical statement of PCA as a least-squares fitting problem.

1933

Harold Hotelling independently redevelops PCA using the covariance matrix eigendecomposition in "Analysis of a Complex of Statistical Variables into Principal Components," giving the method its modern form and name.

1936

Hotelling coins the term latent roots for eigenvalues, and psychologists adopt PCA widely for factor analysis of intelligence test scores.

1963

The Singular Value Decomposition (SVD) algorithm is put on a rigorous numerical footing by Golub and Kahan, making PCA computationally stable for large matrices on digital computers.

1987

Sirovich and Kirby demonstrate PCA on images of faces, showing that human faces can be approximately reconstructed from a small basis of "eigenfaces" — a landmark result in computer vision.

1991

Turk and Pentland publish "Eigenfaces for Recognition," the first real-time PCA-based face recognition system, making PCA famous in the computer vision community.

2000

Kernel PCA is introduced by Schölkopf, Smola, and Müller, extending PCA to nonlinear manifolds via the kernel trick.

2004

Introduction of Randomised PCA (Halko, Martinsson, Tropp) provides approximate but vastly faster PCA for massive datasets — now the default in scikit-learn for large inputs.

2013

Word2Vec embeddings are compressed and visualised using PCA as a standard analysis step, cementing PCA's role as a universal inspection tool throughout the deep learning era.

Why It Matters

Real-World Impact

Face Recognition (Eigenfaces)Turk and Pentland's 1991 system reduced 10,000-pixel face images to ~100 principal components for real-time identification, demonstrating that PCA could compress visual data 100-fold with tolerable quality loss — a technique still used in identity verification pipelines.

Genomics and BioinformaticsPopulation genetic studies routinely run PCA on datasets with hundreds of thousands of genetic markers across thousands of individuals. The first two principal components often map almost exactly onto geographic ancestry, letting researchers detect population structure and correct for it before downstream analysis.

Finance and Algorithmic TradingQuantitative analysts apply PCA to the covariance matrix of asset returns to extract a small number of "risk factors" (e.g., market risk, sector risk, interest rate risk) from hundreds of correlated stock prices, directly powering risk models at firms like BlackRock and Two Sigma.

Image CompressionJPEG is conceptually related to PCA — it projects image patches onto a fixed basis (the Discrete Cosine Transform) that behaves similarly to the empirical principal components of natural images, discarding high-frequency directions that carry little perceptual information.

Data Visualisation and Debugging in ML PipelinesPractitioners routinely project high-dimensional embeddings (e.g., the 768-dimensional outputs of BERT) down to 2D with PCA as a fast sanity-check — if semantically similar inputs don't cluster in the PCA plot, something is wrong with the model or data.

Noise Reduction in Scientific InstrumentsNeuroscience experiments recording from hundreds of electrodes simultaneously use PCA to separate genuine neural signals (high-variance components) from electrical noise and movement artefacts (low-variance components discarded after decomposition).

Recommendation SystemsEarly collaborative filtering systems at Netflix and Amazon applied PCA (or its close cousin, matrix factorisation) to sparse user-item rating matrices to discover "latent taste dimensions" — the conceptual ancestors of modern embedding-based recommenders.

Principal Component Analysis

How It Works

Intuition

Historical Development

Why It Matters

Prerequisites

What This Leads To

See Also

Before this

Where this leads