Neural Networks

Neural networks are parametric function approximators built from layered compositions of weighted sums and nonlinear activation functions, trained by gradient descent on a loss function. Key result: a feedforward network with a single sufficiently large hidden layer can approximate any continuous function on a compact set to arbitrary accuracy (Universal Approximation). Applies in: machine learning, computer vision, natural language processing, function approximation, control. Who/when: McCulloch & Pitts (1943), Rosenblatt (1958), Rumelhart–Hinton–Williams backpropagation (1986), deep learning revival (2006–2012).

A neural network is a function $f_{θ} : R^{n} \to R^{m}$ formed by composing affine maps with elementwise nonlinearities, parameterized by a vector $θ$ of weights and biases. It matters because, given enough parameters and data, such compositions can represent astonishingly rich classes of functions and are trainable by the chain rule (backpropagation) at scale. The central engineering insight is that stacking simple nonlinear units in layers turns a hard representation problem into a tractable optimization problem. A surprising fact: even though the loss landscape of a deep network is highly non-convex with exponentially many critical points, simple stochastic gradient descent reliably finds solutions that generalize — a phenomenon still not fully understood theoretically.

The Core Insight

A single neuron computes a linear decision boundary; it cannot represent nonlinearly separable patterns (e.g., XOR). But if you feed the outputs of many neurons into another layer of neurons, the composition of affine maps and nonlinearities becomes a universal building block. Depth and nonlinearity together convert a limited linear model into a representation that can bend, fold, and carve the input space arbitrarily. Crucially, the whole composite remains differentiable in its parameters, so the gradient of any scalar loss can be computed efficiently by reverse-mode differentiation.

Formal Statement

Artificial Neuron

A single neuron with input $x \in R^{n}$ , weight vector $w \in R^{n}$ , bias $b \in R$ , and activation function $σ : R \to R$ computes:

a = σ (w^{⊤} x + b) = σ (i = 1 \sum n w_{i} x_{i} + b)

⟳model the biological neuron's "weighted summation of inputs followed by a threshold firing rule"; replace the hard threshold with a smooth

σ

for differentiability.

Here $z = w^{⊤} x + b$ is the pre-activation (or logit), and $a$ is the activation (output). The bias $b$ shifts the decision boundary away from the origin.

Feedforward Network / Multilayer Perceptron

A network of $L$ layers maps input $a^{(0)} = x$ through layers indexed $ℓ = 1, \dots, L$ . Let $W^{(ℓ)} \in R^{n_{ℓ} \times n_{ℓ - 1}}$ be the weight matrix and $b^{(ℓ)} \in R^{n_{ℓ}}$ the bias of layer $ℓ$ . Then:

z^{(ℓ)} = W^{(ℓ)} a^{(ℓ - 1)} + b^{(ℓ)}, a^{(ℓ)} = σ^{(ℓ)} (z^{(ℓ)})

⟳stack

n_{ℓ}

neurons of form [1] into one layer; each row of

W^{(ℓ)}

is one neuron's weight vector, and

σ

is applied elementwise.

The full network is the composition $f_{θ} (x) = σ^{(L)} \circ A^{(L)} \circ \dots \circ σ^{(1)} \circ A^{(1)} (x)$ where $A^{(ℓ)} (a) = W^{(ℓ)} a + b^{(ℓ)}$ , and $θ = {W^{(ℓ)}, b^{(ℓ)}}_{ℓ = 1}^{L}$ .

Common Activation Functions

σ_{sig} (z) = \frac{1}{1 + e ^{- z}}, tanh (z) = \frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}}, ReLU (z) = max (0, z)

⟳sigmoid is the logistic CDF squashing

R \to (0, 1)

;

tanh

is a rescaled sigmoid; ReLU is the simplest piecewise-linear nonlinearity,

\partial_{z} ReLU = 1 [z > 0]

Loss Functions

For supervised learning with targets $y$ , common losses for a dataset of $N$ examples are mean squared error (regression) and cross-entropy (classification):

L_{MSE} = \frac{1}{N} k = 1 \sum N ∥ f_{θ} (x_{k}) - y_{k} ∥^{2}, L_{CE} = - \frac{1}{N} k = 1 \sum N c \sum y_{k, c} lo g \overset{p}{^}_{k, c}

⟳MSE is the negative log-likelihood under a Gaussian noise model; cross-entropy is the negative log-likelihood under a categorical (softmax) model.

The softmax output converts logits $z$ into a probability vector: $\overset{p}{^}_{c} = e^{z_{c}} / \sum_{j} e^{z_{j}}$ .

Universal Approximation (Cybenko 1989; Hornik 1991)

Let $σ$ be a continuous, non-constant, bounded function (e.g. sigmoid). For any continuous function $g : K \to R$ on a compact set $K \subset R^{n}$ and any $ε > 0$ , there exist $N \in N$ , weights $w_{i} \in R^{n}$ , biases $b_{i} \in R$ , and coefficients $α_{i} \in R$ such that:

x \in K sup g (x) - i = 1 \sum N α_{i} σ (w_{i}^{⊤} x + b_{i}) < ε

⟳the set of single-hidden-layer functions is dense in

C (K)

; Cybenko's proof uses the Hahn–Banach theorem and discriminatory property of

σ

to show no nonzero signed measure annihilates this set.

That is, a single hidden layer of unbounded width can approximate any continuous function uniformly on compacts. Width may be exponential; depth can give exponentially more efficient representations.

Why It Holds / Derivation

The defining algorithm is backpropagation — efficient computation of $\nabla_{θ} L$ by the chain rule.

Backpropagation

Goal: compute

\partial L / \partial W_{ij}^{(ℓ)}

and

\partial L / \partial b_{i}^{(ℓ)}

for all layers. Step 1 — Define the error signal. Let

δ_{i}^{(ℓ)} = \partial L / \partial z_{i}^{(ℓ)}

, the sensitivity of the loss to the pre-activation of neuron

i

in layer

ℓ

. Collect into

δ^{(ℓ)}

. Step 2 — Output layer. By the chain rule through

a^{(L)} = σ^{(L)} (z^{(L)})

δ^{(L)} = \nabla_{a^{(L)}} L ⊙ σ^{(L)'} (z^{(L)})

⟳

\partial L / \partial z_{i}^{(L)} = (\partial L / \partial a_{i}^{(L)}) (\partial a_{i}^{(L)} / \partial z_{i}^{(L)})

; the second factor is

σ^{'}

;

⊙

is elementwise product.

Step 3 — Recursive backward pass. Since

z_{j}^{(ℓ + 1)} = \sum_{i} W_{j i}^{(ℓ + 1)} a_{i}^{(ℓ)} + b_{j}^{(ℓ + 1)}

and

a_{i}^{(ℓ)} = σ (z_{i}^{(ℓ)})

δ^{(ℓ)} = (W^{(ℓ + 1) ⊤} δ^{(ℓ + 1)}) ⊙ σ^{(ℓ)'} (z^{(ℓ)})

⟳apply the multivariable chain rule

δ_{i}^{(ℓ)} = \sum_{j} δ_{j}^{(ℓ + 1)} (\partial z_{j}^{(ℓ + 1)} / \partial a_{i}^{(ℓ)}) (\partial a_{i}^{(ℓ)} / \partial z_{i}^{(ℓ)})

; the first partial is

W_{j i}^{(ℓ + 1)}

Step 4 — Parameter gradients. Since

z_{i}^{(ℓ)}

depends on

W_{ij}^{(ℓ)}

through

a_{j}^{(ℓ - 1)}

\frac{\partial L}{\partial W _{ij}^{(ℓ)}} = δ_{i}^{(ℓ)} a_{j}^{(ℓ - 1)}, \frac{\partial L}{\partial b _{i}^{(ℓ)}} = δ_{i}^{(ℓ)}

⟳

\partial z_{i}^{(ℓ)} / \partial W_{ij}^{(ℓ)} = a_{j}^{(ℓ - 1)}

and

\partial z_{i}^{(ℓ)} / \partial b_{i}^{(ℓ)} = 1

; multiply by

δ_{i}^{(ℓ)}

Step 5 — Cost. One forward pass plus one backward pass computes the entire gradient in

O (∣ θ ∣)

time — the same order as evaluating the network itself.

■

Gradient Descent Update

Parameters are updated against the gradient with learning rate $η > 0$ . Stochastic gradient descent (SGD) uses a minibatch $B$ :

θ \leftarrow θ - η \nabla_{θ} L_{B} (θ), \nabla_{θ} L_{B} = \frac{1}{∣ B ∣} k \in B \sum \nabla_{θ} ℓ (f_{θ} (x_{k}), y_{k})

⟳first-order Taylor expansion

L (θ - η \nabla) \approx L (θ) - η ∥\nabla ∥^{2}

shows movement along

- \nabla

decreases loss for small

η

; the minibatch estimate is an unbiased gradient estimator.

A two-hidden-layer feedforward network: forward signal flow and backward gradient flow

Frank Rosenblatt | 1928–1971

An American psychologist who, in 1958, built the Perceptron — both an algorithm and a physical machine (the Mark I Perceptron) with motor-driven potentiometers as adjustable weights, designed to recognize images. Rosenblatt proved the Perceptron Convergence Theorem: for linearly separable data, the simple update $w \leftarrow w + y_{k} x_{k}$ on misclassified points terminates in finitely many steps. His optimistic claims drew sharp criticism; Minsky and Papert's 1969 book Perceptrons showed single-layer perceptrons cannot compute XOR, contributing to the first "AI winter." Multilayer networks with backpropagation eventually vindicated the broader research program two decades later.

Building Intuition

Think of each neuron as drawing a hyperplane in input space and outputting "how far, and on which side" (softened by $σ$ ). The first hidden layer carves space into linear regions; the second layer takes logical combinations of those regions (intersections via AND-like patterns, unions via OR-like patterns); deeper layers combine those into ever-more-elaborate shapes. With ReLU, the network is literally a continuous piecewise-linear function — the input space is partitioned into polytopes, and the network is affine on each. Depth multiplies the number of linear regions exponentially, which is why deep narrow networks can be vastly more expressive than shallow wide ones with the same parameter count. Training "bends" all these boundaries simultaneously by following the negative gradient downhill on the loss surface.

Worked Examples

Forward pass through one neuron

Given: input

x = (1, 2)^{⊤}

, weights

w = (0.5, - 1)^{⊤}

, bias

b = 0.5

, activation = sigmoid. Step 1 — Pre-activation:

z = 0.5 (1) + (- 1) (2) + 0.5 = 0.5 - 2 + 0.5 = - 1.0

. Step 2 — Activation:

a = σ (- 1.0) = \frac{1}{1 + e ^{1.0}} = \frac{1}{1 + 2.718} = \frac{1}{3.718} = 0.269

. Step 3 — Derivative (for later backprop):

σ^{'} (z) = a (1 - a) = 0.269 \times 0.731 = 0.197

a = 0.269, σ^{'} (z) = 0.197 (dimensionless)

One backprop / SGD step on MSE

Given: the neuron above with output

a = 0.269

, target

y = 1

, MSE loss

ℓ = (a - y)^{2}

, learning rate

η = 0.1

. Step 1 — Loss gradient w.r.t. output:

\partial ℓ / \partial a = 2 (a - y) = 2 (0.269 - 1) = - 1.462

. Step 2 — Error signal:

δ = \frac{\partial ℓ}{\partial a} σ^{'} (z) = (- 1.462) (0.197) = - 0.288

. Step 3 — Weight gradients:

\partial ℓ / \partial w_{1} = δ x_{1} = - 0.288 (1) = - 0.288

;

\partial ℓ / \partial w_{2} = δ x_{2} = - 0.288 (2) = - 0.576

;

\partial ℓ / \partial b = δ = - 0.288

. Step 4 — Update [9]:

w_{1} \leftarrow 0.5 - 0.1 (- 0.288) = 0.529

;

w_{2} \leftarrow - 1 - 0.1 (- 0.576) = - 0.942

;

b \leftarrow 0.5 - 0.1 (- 0.288) = 0.529

w \to (0.529, - 0.942), b \to 0.529 (moves output toward target 1)

Common Misconceptions

"More layers always help." Naïvely stacking layers can worsen training due to vanishing/exploding gradients: by [7],

δ

is repeatedly multiplied by

W^{⊤}

and

σ^{'}

. With sigmoids (

σ^{'} \leq 0.25

), the gradient shrinks geometrically with depth, so early layers barely learn. Remedies — ReLU activations, careful initialization (He/Xavier), batch normalization, and residual connections — were essential to make deep networks trainable. Depth without these tricks is often harder, not better. "Universal approximation means networks can learn anything from data." Theorem [5] is purely about existence of weights, not learnability. It guarantees nothing about (a) finding those weights by gradient descent, (b) how many samples are required, or (c) generalization to unseen inputs. Approximation, optimization, and generalization are three separate problems.

A single neuron cannot solve XOR

The XOR function maps $(0, 0) \to 0$ , $(0, 1) \to 1$ , $(1, 0) \to 1$ , $(1, 1) \to 0$ . A single neuron computes $σ (w^{⊤} x + b)$ , whose decision boundary $w^{⊤} x + b = 0$ is a single line. No line separates ${(0, 1), (1, 0)}$ from ${(0, 0), (1, 1)}$ — the positive and negative classes are interleaved at opposite corners. Resolution: one hidden layer of two neurons (e.g. computing OR and NAND) followed by an AND output neuron solves XOR exactly. This concrete failure of single-layer perceptrons, highlighted by Minsky–Papert (1969), motivated multilayer networks.

Connections

Neural networks sit at the intersection of several fields. The training objective and update rule come from Gradient Descent and Convex Optimization (though the landscape is non-convex). Backpropagation is exactly reverse-mode Automatic Differentiation applied to a computational graph. The cross-entropy loss and softmax derive from Maximum Likelihood Estimation under a categorical model, linking to Probability Theory. The single-neuron model with sigmoid is precisely Logistic Regression, so a network generalizes a stack of regressions. Universal approximation rests on density arguments from Functional Analysis (Hahn–Banach, Stone–Weierstrass). The matrix operations in [2] are Linear Algebra, and convolutional variants connect to Fourier Analysis and signal processing.

Real-World Applications

Image classification & detection: Convolutional networks (AlexNet 2012, ResNet 2015) power face recognition, medical imaging diagnosis, and autonomous-vehicle perception.
Natural language processing: Transformer networks (BERT, GPT) drive machine translation, search, summarization, and conversational assistants.
Speech recognition: Recurrent and transformer models underlie Siri, Google Assistant, and real-time transcription.
Protein structure prediction: DeepMind's AlphaFold predicts 3-D protein folds from amino-acid sequences, a landmark in computational biology.
Game playing & control: AlphaGo / AlphaZero combine neural value/policy networks with tree search to surpass human champions at Go, chess, and shogi.
Recommendation systems: Netflix, YouTube, and Amazon use deep networks to rank content from user-interaction data.
Generative modeling: Diffusion models and GANs synthesize photorealistic images, audio, and video.

Historical Development

1943 — McCulloch & Pitts publish a logical model of the neuron as a threshold unit, showing networks can compute logical functions.
1949 — Donald Hebb proposes Hebbian learning ("neurons that fire together wire together").
1958 — Rosenblatt introduces the Perceptron and proves its convergence theorem for linearly separable data.
1969 — Minsky & Papert's Perceptrons exposes the XOR limitation; funding and interest decline (first AI winter).
1974/1986 — Werbos (1974) and Rumelhart, Hinton & Williams (1986) popularize backpropagation for training multilayer networks.
1989 — Cybenko proves the Universal Approximation Theorem; LeCun applies convolutional networks to handwritten digits (LeNet).
1997 — Hochreiter & Schmidhuber introduce LSTM, solving long-range dependency learning in recurrent nets.
2012 — Krizhevsky, Sutskever & Hinton's AlexNet wins ImageNet by a wide margin on GPUs, igniting the deep-learning era.
2017 — Vaswani et al. introduce the Transformer ("Attention Is All You Need"), now the dominant architecture for language and beyond.

Subtleties & Limits

Despite practical success, deep learning's theory has large gaps. Why does SGD generalize? Overparameterized networks with more parameters than data points can fit random labels yet still generalize on real data, defying classical bias-variance intuition (the "double descent" phenomenon). Why is non-convex optimization easy? The loss surface has exponentially many saddle points and local minima, yet SGD reliably finds good solutions — partial explanations invoke the geometry of overparameterized landscapes and the "lottery ticket" hypothesis. Expressivity vs. trainability: depth gives exponential expressive power, but provably learnable function classes remain narrow. Robustness and interpretability: networks are vulnerable to imperceptible adversarial perturbations and remain largely black boxes. Sample/energy efficiency: humans learn from far fewer examples and far less energy than current models. These questions span optimization theory, statistical learning theory, and approximation theory.