Activation Functions

Activation functions are mathematical operations applied to the output of each neuron in a neural network, introducing the non-linearity that allows networks to learn complex patterns beyond simple straight-line relationships. Without them, a network with a hundred layers would collapse to the mathematical equivalent of a single layer, making depth meaningless. Remarkably, the choice of a single activation function — swapping sigmoid for ReLU — was one of the key practical breakthroughs that made deep learning viable in the 2010s.

The core insight

A neural network without activation functions is just a fancy linear equation: no matter how many layers you stack, the output is always a weighted sum of the inputs, which can only draw straight lines through data. Activation functions "bend" the space, letting the network carve out curved, complex decision boundaries. Think of each neuron as asking a question — the activation function decides how loudly that neuron "shouts" its answer to the next layer. The whole magic of deep learning depends on this single idea: controlled, differentiable non-linearity.

How It Works

Every neuron computes a weighted sum of its inputs, then passes that sum through the activation function to produce its output. The activation function sits at the boundary between "raw computation" and "what gets passed forward."

1
z=w1x1+w2x2+b\quad z = w_1 x_1 + w_2 x_2 + b

Explanation: This is the raw pre-activation value — a weighted sum of inputs plus a bias, before the activation function touches it.

  • z — the pre-activation value (sometimes called the "logit")
  • w₁, w₂ — learnable weights, one per input
  • x₁, x₂ — the input values coming from the previous layer
  • b — the bias term, a learnable offset
2
a=f(z)\quad a = f(z)

Explanation: The activation function f transforms the raw sum z into the neuron's final output a, which is passed to the next layer.

  • a — the activated output (what the neuron actually "says")
  • f — the chosen activation function (ReLU, sigmoid, tanh, etc.)

The critical requirement is that f must be non-linear (otherwise stacking layers does nothing useful) and differentiable (so gradients can flow backward during training via Backpropagation).

The Major Activation Functions

Sigmoid squashes any value into the range (0, 1), historically used in output layers for binary classification:

3
σ(z)=11+e−z\quad \sigma(z) = \frac{1}{1 + e^{-z}}

Explanation: For very large positive z, σ(z) approaches 1; for very large negative z, it approaches 0 — outputs are always a probability-like value between 0 and 1.

  • e — Euler's number (~2.718), the base of natural logarithm
  • σ(z) — the sigmoid output, always in range (0, 1)

ReLU (Rectified Linear Unit) is the most widely used activation in hidden layers today:

4
ReLU(z)=max⁡(0, z)\quad \text{ReLU}(z) = \max(0,\ z)

Explanation: If the input is positive, pass it straight through; if it is negative or zero, output exactly zero — simple thresholding. ```python import numpy as np import matplotlib.pyplot as plt def sigmoid(z): return 1 / (1 + np.exp(-z)) def relu(z): return np.maximum(0, z) def tanh(z): return np.tanh(z) def leaky_relu(z, alpha=0.01): return np.where(z > 0, z, alpha * z)

# Plot all four z = np.linspace(-4, 4, 200) plt.figure(figsize=(10, 4)) plt.plot(z, sigmoid(z), label='Sigmoid', color='#60a5fa') plt.plot(z, relu(z), label='ReLU', color='#d4a853') plt.plot(z, tanh(z), label='Tanh', color='#a78bfa') plt.plot(z, leaky_relu(z), label='Leaky ReLU', color='#34d399') plt.axhline(0, color='gray', linewidth=0.5) plt.axvline(0, color='gray', linewidth=0.5) plt.legend() plt.title('Common Activation Functions') plt.grid(True, alpha=0.3) plt.show() ```

Tanh is a scaled, shifted sigmoid that outputs values in the range (−1, 1), centering activations around zero:

5
tanh⁡(z)=ez−e−zez+e−z\quad \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}

Explanation: Like sigmoid but symmetric around zero — negative inputs produce negative outputs, positive inputs produce positive outputs, and everything is squashed to (−1, 1). Here is a minimal working example showing how activation functions are used inside a neural network layer with PyTorch: ```python import torch import torch.nn as nn

# A simple two-layer network with different activations model = nn.Sequential( nn.Linear(4, 8), # input layer: 4 features -> 8 neurons nn.ReLU(), # activation after first layer nn.Linear(8, 4), # hidden layer: 8 -> 4 nn.Tanh(), # activation after second layer nn.Linear(4, 1), # output layer: 4 -> 1 nn.Sigmoid() # sigmoid for binary classification output )

# Forward pass x = torch.randn(16, 4) # batch of 16 samples, 4 features each output = model(x) print(output.shape) # torch.Size([16, 1]) ```

Leaky ReLU and the Dead Neuron Fix

Standard ReLU can produce dead neurons — units that always output zero because their weights drift into a region where z is always negative. Leaky ReLU fixes this with a small slope for negative inputs: ```python

# Leaky ReLU in PyTorch — negative inputs get 1% of their value passed through leaky = nn.LeakyReLU(negative_slope=0.01)

# Or the Parametric version (slope is learned during training) prelu = nn.PReLU() ```

researcher · Geoffrey Hinton | b. 1947 | Deep Learning Foundations and Activation Dynamics

Geoffrey Hinton, a British-Canadian cognitive psychologist and computer scientist at the University of Toronto and Google Brain, was instrumental in identifying why sigmoid activations caused the vanishing gradient problem that crippled deep networks — gradients shrink exponentially as they propagate backward through saturating activations. His work on restricted Boltzmann machines, dropout, and advocacy for ReLU-based architectures helped catalyze the 2012 deep learning revolution. In a surprising personal note, Hinton reportedly left Google in 2023 partly to speak more freely about AI risks — one of the field's founding figures becoming one of its most prominent cautionary voices.

Figure
Activation Functions: Input to Output Flow
Neuron Computation: Weighted Sum → Activation → Output x₁ x₂ x₃ w₁ w₂ w₃ Σ +bias z f(z) Activation a out f options: ReLU Sigmoid Tanh Softmax Leaky ReLU
Figure
ReLU vs Sigmoid: Shape and Gradient Comparison
ReLU vs Sigmoid — Shape and Saturation ReLU: max(0, z) grad=0 grad=1 z a -4 0 +4 dead zone Sigmoid: 1/(1+e⁻ᶻ) saturates→0 saturates→1 active z a -4 0 +4 output: (0,1)

Intuition

The Mental Model

Imagine you are hiring a panel of judges to score a talent competition. Each judge (neuron) listens to all the performers and forms a raw opinion — a number that could be anywhere from hugely negative to enormously positive. But the show needs the judges to communicate clearly and consistently, so each judge runs their opinion through a personal "response style" (the activation function) before announcing it. One judge uses a light switch style (ReLU): if their raw feeling is positive, they say exactly how positive they feel; if it's negative, they say nothing at all — zero. This is fast, honest, and easy to understand, but sometimes a judge goes completely silent and never speaks again (the dead neuron problem). Another judge uses a dimmer switch style (Sigmoid): they always give a response between 0% and 100% enthusiasm. Useful when you need a probability, but extreme opinions get squashed near the edges — making it hard to tell a "very good" act from an "insanely good" one. A third judge uses hot-and-cold style (Tanh): like the dimmer switch, but they can express active dislike (negative numbers), centering their feedback around neutral zero. The key insight is this: without these response styles, all judges would just be passing raw numbers forward, and stacking more judges would never add any new information — they'd all just be restating the same linear sum. The "bending" introduced by each judge's style is what lets the whole panel together recognise subtlety and complexity that no single judge could catch alone.

Historical Development

Origins · 1 minhow this was discovered
History
1943
McCulloch & Pitts introduce the first mathematical model of a neuron — a binary threshold unit using a simple step function as its activation, either firing or not firing.
1958
Frank Rosenblatt develops the Perceptron, using a step activation function, and demonstrates learning on simple classification tasks.
1974
Paul Werbos describes backpropagation in his PhD thesis, laying the groundwork for training multi-layer networks — but requiring smooth (differentiable) activations, something the step function lacks.
1986
Rumelhart, Hinton, and Williams publish the landmark backpropagation paper, using differentiable sigmoid activations and demonstrating practical multi-layer learning for the first time at scale.
1991–1998
Researchers including Hochreiter and Bengio formally characterise the vanishing gradient problem — sigmoid and tanh activations cause gradients to shrink exponentially in deep networks, stalling learning.
2000
Hahnloser et al. publish a neuroscience-inspired paper introducing the rectified linear unit (ReLU) in a computational context, noting its biological plausibility in neurons that only fire when excited.
2011
Glorot, Bordes, and Bengio publish "Deep Sparse Rectifier Neural Networks," rigorously demonstrating that ReLU trains faster and achieves better results than sigmoid/tanh — catalysing a field-wide shift to ReLU as the default.
2012
Krizhevsky, Sutskever, and Hinton's AlexNet wins ImageNet using ReLU activations at scale, proving the real-world power of this choice and cementing ReLU as the standard hidden-layer activation.
2017–2020
Research into advanced activations accelerates — Hendrycks & Gimpel introduce GELU (used in BERT and GPT), and Ramachandran et al. propose Swish (later used in EfficientNet), showing learned or smoother activations can outperform ReLU in certain architectures.

Why It Matters

Real-World Impact
GPT and Large Language Models (GELU)GPT-2, GPT-3, and BERT all use the GELU (Gaussian Error Linear Unit) activation in their feed-forward sub-layers, chosen because it produces smoother gradients and slightly better language modelling performance compared to ReLU in transformer architectures.
AlexNet and Modern CNNs (ReLU)The 2012 AlexNet architecture was among the first at scale to use ReLU throughout all convolutional layers, cutting training time by ~6× compared to tanh equivalents on the same hardware — a directly attributable speedup that changed industry practice overnight.
Binary Classifiers — Spam/Fraud Detection (Sigmoid Output)Logistic regression and single-output neural network classifiers used in email spam filters (Gmail) and credit card fraud detection (Stripe, PayPal) use sigmoid as the final activation to produce a calibrated probability score between 0 and 1.
Multi-class Image Classification (Softmax Output)ImageNet classifiers, Google Lens, and medical imaging models use Softmax as their output activation to produce a probability distribution across all possible classes, ensuring all class probabilities sum to 1.
EfficientNet / MobileNet (Swish/ReLU6)Google's EfficientNet family, widely deployed on mobile devices and in Google Photos, uses Swish activations and the bounded ReLU6 variant to improve accuracy under tight computational budgets.
Recurrent Networks and LSTMs (Tanh + Sigmoid)The LSTM cell — central to translation systems (early Google Translate), voice assistants, and time-series forecasting — uses tanh to control cell state magnitude and sigmoid gates to control information flow, a carefully designed combination.
Generative Models — GANs (LeakyReLU)DCGAN and StyleGAN architectures use Leaky ReLU in the discriminator network, avoiding dead neurons and keeping gradients alive when the discriminator needs to learn from both real and fake examples during adversarial training.

Prerequisites

What This Leads To

See Also

Neural Networks · Backpropagation · Loss Functions · Vanishing and Exploding Gradients · Batch Normalization

Before this

Where this leads