Activation Functions: Sigmoid, Tanh, ReLU

An activation function $\phi: \mathbb{R} \to \mathbb{R}$ is applied elementwise to the pre-activation $z = w^\top x + b$ produced by a neuron, giving the neuron's output $a = \phi(z)$. Without $\phi$ (or with $\phi$ the identity), a network of $L$ layers computes

which is an affine function of $x$: expand the products and you get $a^{(L)} = W' x + b'$ for some single matrix $W'$ and vector $b'$. Depth added zero expressive power. This is the precise reason nonlinearity is necessary, not merely conventional: composing linear maps yields a linear map, but composing nonlinear maps yields a much richer function class, which is the content later formalized by the The Universal Approximation Theorem. Three functions dominate the historical and practical landscape. Sigmoid (logistic function) $\sigma(z) = \frac{1}{1+e^{-z}}$ maps $\mathbb{R} \to (0,1)$ and was the original biologically-motivated choice, since it mimics a neuron's firing rate saturating at 0 or 1. Tanh $\tanh(z) = \frac{e^{z}-e^{-z}}{e^{z}+e^{-z}} = 2\sigma(2z)-1$ maps $\mathbb{R} \to (-1,1)$ and is simply a rescaled, shifted sigmoid — but the rescaling matters enormously because tanh is zero-centered while sigmoid is not. ReLU (Rectified Linear Unit) $\text{ReLU}(z) = \max(0,z)$ abandons smoothness and boundedness entirely in favor of computational simplicity and a derivative that does not shrink for positive inputs. The property that separates these functions in practice is saturation: a region where the function is nearly flat, so its derivative is near zero. Sigmoid and tanh saturate on both sides ($z \to \pm\infty$); ReLU saturates only on the negative side (it is exactly zero, with zero gradient, for all $z < 0$) and is unbounded and gradient-preserving on the positive side. This asymmetry is the single fact that explains most of the empirical differences between the three functions.

Step 1 — Compute the pre-activation. Each neuron computes $z = w^\top x + b$, a linear combination of inputs (the mechanism already described in The Artificial Neuron).

Step 2 — Apply the nonlinearity elementwise. Choose $\phi \in \{\sigma, \tanh, \text{ReLU}\}$ and compute $a = \phi(z)$. In a layer with many units, this is applied coordinate-wise to a vector.

Step 3 — Forward propagate. The output $a$ becomes the input to the next layer, repeating steps 1–2 across all layers — this is exactly the forward pass of a The Multilayer Perceptron.

Step 4 — Backpropagate through the nonlinearity. During training, the chain rule requires $\phi'(z)$, the local derivative, at every unit. This value multiplies the upstream gradient flowing backward, so its magnitude directly controls how much gradient signal survives — this is the mechanism at the heart of the The Vanishing and Exploding Gradient Problem. The three derivatives, which you should be able to derive from the definitions:

Sigmoid's derivative is derived from the quotient rule on $\sigma(z) = (1+e^{-z})^{-1}$: $\sigma'(z) = e^{-z}(1+e^{-z})^{-2} = \sigma(z)(1-\sigma(z))$, confirmed by substitution. Tanh's identity follows similarly from $\frac{d}{dz}\frac{e^z - e^{-z}}{e^z+e^{-z}}$ via the quotient rule, simplifying to $1-\tanh^2(z)$. ``` function FORWARD_LAYER(x, W, b, activation): z = W · x + b # pre-activation, vector of size n if activation == "sigmoid": a = 1 / (1 + exp(-z)) # elementwise else if activation == "tanh": a = (exp(z) - exp(-z)) / (exp(z) + exp(-z)) else if activation == "relu": a = max(0, z) # elementwise return a, z # cache z for the backward pass function BACKWARD_LAYER(dA, z, activation):

# dA: upstream gradient dL/dA, same shape as z if activation == "sigmoid": s = 1 / (1 + exp(-z)) dPhi = s (1 - s) else if activation == "tanh": t = (exp(z) - exp(-z)) / (exp(z) + exp(-z)) dPhi = 1 - t t else if activation == "relu": dPhi = (z > 0) ? 1 : 0 # subgradient convention at z = 0 dZ = dA * dPhi # elementwise product (chain rule) return dZ # pass on to compute dW, db, and dX ``` The pseudocode is deliberately structured to mirror real autodiff frameworks: `FORWARD_LAYER` caches $z$ because `BACKWARD_LAYER` needs it, and the backward step is always "multiply the incoming gradient by the local derivative," which is the elementary chain-rule step repeated at every layer during backpropagation.

Time complexity. Each activation is applied elementwise, so for a layer with $n$ units the forward cost is $O(n)$ evaluations of $\phi$, and the backward cost is $O(n)$ evaluations of $\phi'$. The constant factor differs meaningfully: sigmoid and tanh require an exponential $e^{-z}$ or $e^{2z}$ (a transcendental function, typically 10–20× the cost of a comparison), while ReLU requires only a comparison against zero, `max(0, z)`. Across a network with $L$ layers and total width $N = \sum_\ell n_\ell$, the total activation cost is $O(N)$ per forward or backward pass — asymptotically the same order as the matrix multiplications ($O(\sum_\ell n_\ell n_{\ell-1})$), but with ReLU's tiny constant it is usually negligible next to the matmuls, whereas sigmoid/tanh's exponential evaluations can be a measurable fraction of runtime on large, wide layers.

Space complexity. Backpropagation must cache the pre-activation $z$ (or equivalently $a$, since $\phi'$ can often be written in terms of $a$: $\sigma'(z) = a(1-a)$, $\tanh'(z) = 1-a^2$) for every unit in every layer, since the backward pass needs it. This costs $O(N)$ additional memory per example, matching the memory footprint of storing activations generally — it is not activation-specific overhead but a property of backpropagation itself.

Correctness of the derivative computations. The chain rule guarantees that if $a = \phi(z)$ and $z = z(w)$, then $\frac{\partial L}{\partial z} = \frac{\partial L}{\partial a}\cdot \phi'(z)$, which is exactly the multiplication `dA * dPhi` in the pseudocode. The correctness of each closed-form derivative follows from ordinary single-variable calculus (quotient rule, as sketched above), and can be double-checked numerically: for any $z$, $\frac{\phi(z+\epsilon) - \phi(z-\epsilon)}{2\epsilon} \to \phi'(z)$ as $\epsilon \to 0$ (central finite differences), which is the standard gradient-checking technique used to catch bugs in hand-derived backward passes.

Why nonlinearity is necessary — the collapse argument, formally. Suppose every layer uses $\phi = \text{identity}$. Then layer $\ell$ computes $a^{(\ell)} = W_\ell a^{(\ell-1)} + b_\ell$. Substituting recursively:

for some vector $b'$ built from the $b_\ell$'s and the $W_\ell$'s. Since $W'$ is a single matrix and $b'$ a single vector, the entire $L$-layer network is exactly one affine map — no more expressive than a single-layer perceptron, regardless of how large $L$ or the hidden widths are. Any nonlinear, non-affine $\phi$ breaks this collapse, since $\phi(Wx+b)$ is generically not affine in $x$; this is the minimal condition the The Universal Approximation Theorem requires of an activation function.

Tradeoffs, stated honestly.

• Sigmoid/tanh are smooth and bounded, giving stable, interpretable outputs, but $\phi'(z) \to 0$ as $|z| \to \infty$ means gradients shrink multiplicatively across layers — the core mechanism of vanishing gradients.
• ReLU avoids saturation on the positive side ($\phi'=1$ exactly, no shrinkage), which is why it trains deep networks far more reliably, but a unit with $z<0$ forever (e.g., after a large negative weight update) has zero gradient forever and can "die," never updating again — an asymmetric failure mode sigmoid/tanh do not have in the same way.
• ReLU is not differentiable at $z=0$; frameworks sidestep this by defining a subgradient (0 or 1), which is mathematically imprecise but practically harmless since $z=0$ has measure zero under continuous input distributions.

Consider a single hidden unit with weight $w = 2$, bias $b = -1$, receiving scalar input $x = 0.6$, so the pre-activation is $z = w x + b = 2(0.6) - 1 = 0.2$. We compute the forward output and the local derivative under each activation, then propagate an upstream gradient of $\delta = \frac{\partial L}{\partial a} = 0.5$ (say, coming from a downstream loss) back to $z$.

Now push $z$ into the negative region by changing the input to $x = -0.9$, giving $z = 2(-0.9) - 1 = -2.8$:

This side-by-side trace makes the saturation phenomenon concrete: at $z=0.2$ (near zero) all three gradients are of reasonable magnitude, but at $z=-2.8$ sigmoid and tanh gradients have shrunk to nearly nothing (0.054 and 0.0147 respectively), and ReLU's gradient is exactly zero — this unit is now "dead" for this input and contributes no learning signal at all through this path, illustrating both the vanishing-gradient behavior of the saturating functions and the dying-ReLU phenomenon in one worked comparison.

The mathematical shapes of sigmoid and tanh predate neural networks by over a century — the logistic function was studied by Pierre François Verhulst in the 1830s for population growth — but their adoption as neuron nonlinearities is tied to the perceptron and connectionist tradition. Frank Rosenblatt's perceptron (1958) used a hard threshold, not a smooth sigmoid; smooth, differentiable activations became essential once gradient-based training via backpropagation required a well-defined derivative everywhere. Sigmoid and tanh dominated neural network research through the 1980s and 1990s precisely because they are differentiable substitutes for the threshold units of earlier perceptron models.

history 1958 — Rosenblatt's perceptron uses a hard-threshold nonlinearity, the conceptual ancestor of later smooth activations. 1986 — Rumelhart, Hinton, and Williams popularize backpropagation, driving widespread use of differentiable sigmoid and tanh units. 2000 — Hahnloser, Sarpeshkar, Mahowald, Douglas, and Seung describe rectified linear-like dynamics in biological neural circuit models. 2010 — Nair and Hinton show ReLU improves training of restricted Boltzmann machines and deep networks over sigmoid/tanh units. 2012 — AlexNet uses ReLU throughout a deep convolutional network, contributing to a large ImageNet accuracy jump and cementing ReLU as the default hidden-layer nonlinearity.

Activation functions are not a historical footnote — they are a live design choice in every network trained today. Convolutional networks for image classification (ResNet, EfficientNet, and their descendants) use ReLU or its smoother relatives (Leaky ReLU, GELU, Swish) almost universally in hidden layers because non-saturating gradients let gradient signal survive tens or hundreds of layers. Recurrent architectures such as LSTMs and GRUs still rely heavily on sigmoid and tanh internally: sigmoid gates control what information to keep or forget (its $(0,1)$ range naturally represents a "how much" fraction), while tanh squashes candidate cell-state updates into a bounded, zero-centered range that keeps hidden states numerically stable across long sequences. Output layers for binary classification use sigmoid to produce a probability in $(0,1)$; multi-class classifiers generalize this with softmax, a multivariate sibling of sigmoid. Transformer architectures, the backbone of large language models, typically use GELU or Swish — smooth cousins of ReLU designed to keep ReLU's gradient-preserving benefits while avoiding its exact-zero dead zone. Any framework you use — PyTorch, TensorFlow, JAX — ships all three functions as primitive, hardware-accelerated operations, because they sit on the critical path of essentially every forward and backward pass executed in deep learning.

Ask the tutor: How does the choice between sigmoid, tanh, and ReLU concretely change what happens during backpropagation in a 50-layer network?