Vectors and Matrices in Machine Learning

Vectors and matrices are the fundamental data structures of machine learning: a vector is an ordered list of numbers encoding a single example or parameter set, and a matrix is a rectangular array that stores datasets, linear transformations, and learnable weights. Key result in words: nearly every ML model — from linear regression to deep neural networks — is built by repeatedly applying matrix-vector multiplication, transposition, and inner products, making linear algebra the computational backbone of the field. Where it applies: feature representation, regression, classification, dimensionality reduction, neural networks, recommendation systems. Who/when: linear algebra formalized in the 19th–20th centuries (Cayley, Sylvester); adapted to statistical learning from the 1950s onward and to modern deep learning from the 2010s.

A vector $x \in R^{n}$ is an ordered $n$ -tuple of real numbers, geometrically a point or arrow in $n$ -dimensional space; a matrix $A \in R^{m \times n}$ is a 2D array of numbers that represents either a collection of vectors or a linear map between vector spaces. In machine learning, datasets are stored as matrices (rows = examples, columns = features), model parameters are vectors or matrices, and the act of computing a prediction is almost always a matrix-vector product followed by a nonlinearity. This matters because casting learning as linear algebra unlocks decades of numerical methods, allows massive parallelism on GPUs, and makes gradients computable through clean matrix calculus. A surprising fact: even though neural networks are famously "nonlinear," more than 99% of their floating-point operations are pure linear algebra (matrix multiplies), with nonlinearity supplied by cheap element-wise functions in between.

The Core Insight

Data, parameters, and transformations all live in the same algebraic language. A learning algorithm reduces to: stack your data into a matrix

X

, store your model in a parameter vector/matrix

W

, compute predictions via

X W

, measure error, and adjust

W

using gradients that are themselves matrix expressions. Because matrix multiplication composes linear maps, stacking layers is just multiplying matrices, and this single operation — generalized inner product — is the atom from which most of ML is assembled.

Formal Statement

Vector

A vector in $R^{n}$ is an ordered tuple

x = x_{1} x_{2} ⋮ x_{n}, x_{i} \in R .

⟳

R^{n}

is the

n

-fold Cartesian product of

R

with itself, equipped with componentwise addition and scalar multiplication.

Here $n$ is the dimension (number of features), $x_{i}$ is the $i$ -th component, and by convention vectors are columns. Each $x$ represents one example, e.g. a flower described by 4 measurements gives $x \in R^{4}$ .

Matrix

A matrix $A \in R^{m \times n}$ is a rectangular array

A = a_{11} a_{21} ⋮ a_{m 1} a_{12} a_{22} ⋮ a_{m 2} \dots \dots ⋱ \dots a_{1 n} a_{2 n} ⋮ a_{mn},

⟳a matrix is a function

{1, \dots, m} \times {1, \dots, n} \to R

(i, j) \mapsto a_{ij}

, displayed as a grid.

with $m$ rows and $n$ columns; $a_{ij}$ is the entry in row $i$ , column $j$ . In a design matrix $X \in R^{m \times n}$ , each of the $m$ rows is one example and each of the $n$ columns is one feature.

Core operations

The inner (dot) product, matrix-vector product, and matrix-matrix product are

a^{⊤} b = i = 1 \sum n a_{i} b_{i}, (A x)_{i} = j = 1 \sum n a_{ij} x_{j}, (A B)_{ik} = j = 1 \sum n a_{ij} b_{j k} .

⟳each output entry is the dot product of a row of the left operand with a column of the right operand; sizes must conform

(m \times n) (n \times p) = (m \times p)

The transpose $A^{⊤}$ has entries $(A^{⊤})_{ij} = a_{j i}$ . The Euclidean norm is $∥ x ∥_{2} = x^{⊤} x$ .

Linear maps are matrices

Every linear map $T : R^{n} \to R^{m}$ (satisfying $T (α u + β v) = α T (u) + β T (v)$ ) is represented by a unique matrix $A \in R^{m \times n}$ such that

T (x) = A x, A = [T (e_{1}) T (e_{2}) \dots T (e_{n})] .

⟳write

x = \sum_{j} x_{j} e_{j}

, apply linearity, and read off

T (x) = \sum_{j} x_{j} T (e_{j})

; the columns of

A

are the images of the standard basis vectors.

Why It Holds / Derivation

Matrix multiplication = composition of linear maps

We show that multiplying matrices computes the composition of the maps they represent — the fact that makes stacking neural-network layers equivalent to multiplying weight matrices.

Let $T_{B} : R^{p} \to R^{n}$ have matrix $B \in R^{n \times p}$ and $T_{A} : R^{n} \to R^{m}$ have matrix $A \in R^{m \times n}$ . The composition $T_{A} \circ T_{B} : R^{p} \to R^{m}$ is linear (composition of linear maps is linear).
By [4], $T_{A} \circ T_{B}$ has a matrix $C$ whose $k$ -th column is $(T_{A} \circ T_{B}) (e_{k}) = T_{A} (B e_{k}) = A (B e_{k})$ .
Now $B e_{k}$ is the $k$ -th column of $B$ , call it $b_{k}$ , with entries $b_{j k}$ . Then $A b_{k}$ has $i$ -th entry $\sum_{j = 1}^{n} a_{ij} b_{j k}$ .
Therefore $C_{ik} = \sum_{j = 1}^{n} a_{ij} b_{j k}$ , which is exactly the definition $(A B)_{ik}$ in [3]. Hence $C = A B$ and $(T_{A} \circ T_{B}) (x) = A B x$ .
Consequence: a two-layer linear network $x \mapsto A (B x)$ collapses to a single matrix $A B$ — which is why nonlinearities between layers are essential. $■$

Normal equations for linear regression

We derive the closed-form least-squares solution, the canonical ML use of these objects.

Minimize the squared error $L (w) = ∥ X w - y ∥_{2}^{2}$ where $X \in R^{m \times n}$ , $y \in R^{m}$ .
Expand: $L = (X w - y)^{⊤} (X w - y) = w^{⊤} X^{⊤} X w - 2 y^{⊤} X w + y^{⊤} y$ .
Take the gradient with respect to $w$ : $\nabla_{w} L = 2 X^{⊤} X w - 2 X^{⊤} y$ .
Set to $0$ : $X^{⊤} X w = X^{⊤} y$ (the normal equations).
If $X^{⊤} X$ is invertible, $w^{⋆} = (X^{⊤} X)^{- 1} X^{⊤} y$ . $■$

w^{⋆} = (X^{⊤} X)^{- 1} X^{⊤} y .

⟳differentiate the quadratic form using

\nabla_{w} (w^{⊤} M w) = (M + M^{⊤}) w

and

\nabla_{w} (c^{⊤} w) = c

, with

M = X^{⊤} X

symmetric.

Arthur Cayley | 1821–1895

Cayley introduced matrices as standalone algebraic objects in his 1858 Memoir on the Theory of Matrices, defining matrix multiplication so that it mirrored the composition of linear substitutions — the exact property exploited in deep-network layer stacking. He proved the Cayley–Hamilton theorem (a matrix satisfies its own characteristic equation), establishing matrices as a self-contained algebra rather than mere notation. His collaborator James Joseph Sylvester coined the word "matrix" (Latin for "womb") in 1850. Together they transformed array notation into the algebraic machinery that, a century later, would underpin statistical learning and neural computation.

Building Intuition

Think of a vector as an arrow from the origin: its components are how far you walk along each coordinate axis. A matrix is a machine that bends space — it takes every arrow and maps it to a new arrow, possibly rotating, stretching, shearing, or projecting. Crucially, this machine is fully described by what it does to the basis arrows $e_{1}, \dots, e_{n}$ (the columns of the matrix). In ML, the dot product $w^{⊤} x$ measures alignment: it is large and positive when the data point $x$ points in the same direction as the weight vector $w$ , which is exactly how a linear classifier decides "yes" vs "no." A whole dataset is a cloud of points (a matrix), and learning is the act of finding directions $w$ in which the cloud separates, aligns, or varies most.

A 2×2 matrix transforming the unit square: the columns of A are the images of the basis vectors e₁ and e₂.

Worked Examples

Predicting house price with a dot product

Given: a weight vector

w = [150, - 2000, 30000]^{⊤}

(price per sq-ft, per mile from city, per bedroom), bias

b = 50000

, and a house

x = [1200, 5, 3]^{⊤}

(sq-ft, miles, bedrooms). Steps:

Inner product (using [3]):

w^{⊤} x = 150 (1200) + (- 2000) (5) + 30000 (3)

$= 180000 - 10000 + 90000 = 260000$ .
Add bias: $\overset{y}{^} = w^{⊤} x + b = 260000 + 50000$ .

Answer:

\overset{y}{^} = $310, 000

The single dot product is the entire prediction of a linear regression model.

Batch predictions and least squares

Given: design matrix and targets

X = 111123, y = 224

(first column = bias, second = a feature). Fit

w = [w_{0}, w_{1}]^{⊤}

via [5]. Steps:

$X^{⊤} X = [36614]$ , with $det = 3 (14) - 36 = 6$ .
$(X^{⊤} X)^{- 1} = \frac{1}{6} [14 - 6 - 6 3]$ .
$X^{⊤} y = [1 + 1 + 1 1 + 2 + 3] \cdot y$ → compute: $[2 + 2 + 4 1 (2) + 2 (2) + 3 (4)] = [818]$ .
$w^{⋆} = \frac{1}{6} [14 - 6 - 6 3] [818] = \frac{1}{6} [112 - 108 - 48 + 54] = \frac{1}{6} [46]$ .

Answer:

w_{0} = \frac{2}{3} \approx 0.667, w_{1} = 1

The fitted line is

\overset{y}{^} = 0.667 + x

(dimensionless prediction matching

y

best in squared error).

Common Misconceptions

Matrix multiplication is NOT commutative

In general $A B \neq = B A$ — and often one product is undefined even when the other exists (shapes must conform). Order encodes which transformation happens first. A frequent ML bug is writing $x^{⊤} W$ when you mean $W x$ , flipping the meaning of rows and columns. Also: $(A B)^{⊤} = B^{⊤} A^{⊤}$ (the order reverses), not $A^{⊤} B^{⊤}$ .

Stacking linear layers gains nothing without nonlinearity

Consider two "layers" $h = B x$ then $z = A h$ . One might hope a deeper network is more expressive. But $z = A (B x) = (A B) x = C x$ for a single matrix $C = A B$ . So any number of stacked linear layers is equivalent to one linear layer — it represents exactly the same set of functions. This is precisely why neural networks insert nonlinear activations (ReLU, sigmoid) between matrix multiplies; without them, depth is an illusion.

Singular $X^\top X$ breaks the normal equations

Equation [5] requires $X^{⊤} X$ to be invertible, which fails when features are collinear or when $n > m$ (more features than examples). In practice one adds regularization: $w^{⋆} = (X^{⊤} X + λ I)^{- 1} X^{⊤} y$ (ridge regression), guaranteeing invertibility for any $λ > 0$ , or uses the pseudoinverse / gradient descent. Never blindly invert a matrix in code — solve the linear system instead, for numerical stability.

Connections

Vectors and matrices are the substrate for nearly every neighbouring topic. Eigenvalues and Eigenvectors reveal the invariant directions of a matrix and power Principal Component Analysis for dimensionality reduction. The Singular Value Decomposition factorizes any matrix and underlies recommendation systems and low-rank approximation. Gradient Descent navigates parameter space using gradient vectors, and Matrix Calculus gives the rules for differentiating the vector/matrix expressions appearing in Backpropagation. The geometry of Inner Product Spaces formalizes the dot products used by Support Vector Machines and kernel methods, while Norms and Distances define the loss functions being minimized.

Real-World Applications

Linear & logistic regression — predictions are $X w$ (a matrix-vector product); used in credit scoring, epidemiology, and A/B testing.
Deep neural networks — each dense layer computes $σ (W x + b)$ ; GPUs/TPUs exist primarily to run these matrix multiplies fast (e.g. GPT-class transformers).
Principal Component Analysis (PCA) — the covariance matrix's eigenvectors compress high-dimensional data (face recognition, gene expression).
Recommendation systems — Netflix/Spotify factor a huge user-item matrix $R \approx U V^{⊤}$ to predict ratings (matrix factorization, SVD).
Computer graphics & vision — image rotations, projections, and convolutions are matrix operations; convolutional layers reduce to matrix multiplies via im2col.
Word embeddings & transformers — words become vectors; attention computes $softmax (Q K^{⊤}) V$ , three matrix products at the heart of large language models.
PageRank — the web's link structure is a giant matrix whose dominant eigenvector ranks pages (the original Google algorithm).

Historical Development

1843 — Hamilton invents quaternions, an early non-commutative system foreshadowing matrix algebra.
1850 — Sylvester coins the term "matrix."
1858 — Cayley's Memoir on the Theory of Matrices defines matrix multiplication and the algebra of matrices.
1888 — Peano gives the modern axiomatic definition of a vector space.
1900s — Hilbert and others develop infinite-dimensional vector (Hilbert) spaces, foundational for functional analysis and later kernel methods.
1936 — Eckart and Young prove the low-rank approximation theorem via SVD, basis of modern data compression.
1958 — Rosenblatt's perceptron computes $sign (w^{⊤} x)$ , framing learning as finding a weight vector.
1986 — Rumelhart, Hinton & Williams popularize backpropagation, expressing learning as matrix-calculus gradient flow.
2012 — AlexNet exploits GPU matrix multiplication to win ImageNet, igniting the deep-learning era; matrix multiply becomes the dominant compute primitive of AI.

Subtleties & Limits

The relentless growth of model sizes makes the cost of matrix multiplication a central bottleneck and open research frontier. The asymptotic complexity of multiplying two $n \times n$ matrices is still unknown: the naïve algorithm is $O (n^{3})$ , Strassen (1969) achieved $O (n^{2.807})$ , and the current record (Williams et al.) is below $O (n^{2.372})$ , yet the true exponent $ω \geq 2$ remains undetermined. Practically, these fast algorithms are numerically delicate and rarely used in ML, where hardware-optimized $O (n^{3})$ kernels dominate. Parallel open questions include: how to compress weight matrices via low-rank or sparse structure without losing accuracy, how to multiply huge matrices in low precision (8-bit, 4-bit) stably, and how to schedule matrix operations across thousands of accelerators. The mathematics of which matrices ML actually needs (highly structured, low effective rank) versus the worst case studied in theory is itself an active area.