ReLU and Its Variants

ReLU (Rectified Linear Unit) is an activation function that outputs the input directly if it is positive, and zero otherwise — a deceptively simple rule that transformed deep learning by making very deep networks trainable. It matters because nearly every modern neural network, from image classifiers to large language models, uses ReLU or one of its variants as the nonlinearity between layers. The counterintuitive fact: a function that is literally just "ignore negatives" outperforms smooth, mathematically elegant alternatives like sigmoid and tanh in most practical deep learning tasks.

The core insight

A neural network without activation functions is just a fancy linear regression — no matter how many layers you stack, it collapses into a single matrix multiplication. Activation functions introduce the nonlinearity that lets networks learn complex patterns, and ReLU does this in the simplest possible way: pass positive values through unchanged, kill negative values to zero. The "aha moment" is realizing that this brutal simplicity is a feature, not a bug — ReLU trains faster, avoids the vanishing gradient problem that crippled earlier networks, and is trivially cheap to compute. Everything else in this article is a refinement of this one idea.

How It Works

ReLU is defined by a single rule: if the input is positive, return it; otherwise return zero.

1
ReLU(x)=max⁡(0, x)\quad \text{ReLU}(x) = \max(0,\, x)

Explanation: For any input value x, take whichever is larger — x itself or zero — discarding the negative portion entirely. During training, networks learn by computing gradients (how much to adjust each weight). ReLU's gradient is simple:

2
ddxReLU(x)={1if x>00if x≤0\quad \frac{d}{dx}\text{ReLU}(x) = \begin{cases} 1 & \text{if } x > 0 \\ 0 & \text{if } x \leq 0 \end{cases}

Explanation: Gradient flows through unchanged when the neuron is active (x > 0), and is completely blocked when the neuron is inactive (x ≤ 0). Leaky ReLU fixes the dying neuron problem by allowing a small slope α (typically 0.01) on the negative side:

3
LeakyReLU(x)={xif x>0αxif x≤0\quad \text{LeakyReLU}(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha x & \text{if } x \leq 0 \end{cases}

Explanation: Instead of zeroing out negative inputs, multiply them by a small constant α so gradients can still flow backward through "inactive" neurons. Here is the entire ReLU family implemented in plain Python so you can see exactly what each variant does: ```python import math def relu(x): """Standard ReLU: max(0, x)""" return max(0, x) def leaky_relu(x, alpha=0.01): """Leaky ReLU: small slope for negatives""" return x if x > 0 else alpha x def elu(x, alpha=1.0): """Exponential Linear Unit: smooth negative region""" return x if x > 0 else alpha (math.exp(x) - 1) def gelu(x): """Gaussian Error Linear Unit: used in Transformers""" return x 0.5 (1 + math.tanh(math.sqrt(2 / math.pi) (x + 0.044715 x**3)))

# Try them all on a range of values test_values = [-2.0, -0.5, 0.0, 0.5, 2.0] for v in test_values: print(f"x={v:5.1f} | ReLU={relu(v):.3f} | " f"Leaky={leaky_relu(v):.3f} | " f"ELU={elu(v):.3f} | " f"GELU={gelu(v):.3f}") ``` ``` x= -2.0 | ReLU=0.000 | Leaky=-0.020 | ELU=-0.865 | GELU=-0.045 x= -0.5 | ReLU=0.000 | Leaky=-0.005 | ELU=-0.393 | GELU=-0.154 x= 0.0 | ReLU=0.000 | Leaky= 0.000 | ELU= 0.000 | GELU= 0.000 x= 0.5 | ReLU=0.500 | Leaky= 0.500 | ELU= 0.500 | GELU= 0.346 x= 2.0 | ReLU=2.000 | Leaky= 2.000 | ELU= 2.000 | GELU= 1.954 ``` And here is how you use ReLU in a real PyTorch model: ```python import torch import torch.nn as nn

# Option 1: As a standalone function x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0]) print(torch.relu(x))

# tensor([0., 0., 0., 1., 2.])

# Option 2: As a layer in a Sequential model model = nn.Sequential( nn.Linear(784, 256), # Input layer nn.ReLU(), # Standard ReLU activation nn.Linear(256, 128), # Hidden layer nn.ReLU(), # Another ReLU nn.Linear(128, 10), # Output layer (no activation — raw logits) )

# Option 3: Using variants model_leaky = nn.Sequential( nn.Linear(784, 256), nn.LeakyReLU(negative_slope=0.01), # Leaky ReLU nn.Linear(256, 10), ) model_gelu = nn.Sequential( nn.Linear(784, 256), nn.GELU(), # Used in BERT, GPT-2/3/4 nn.Linear(256, 10), ) ```

researcher · Geoffrey Hinton | b. 1947 | Backpropagation and Deep Network Training

Geoffrey Hinton is a British-Canadian cognitive scientist and computer scientist who spent decades at the University of Toronto and Google Brain before his high-profile departure from Google in 2023 to speak freely about AI risks. His work on backpropagation in the 1980s laid the groundwork that makes ReLU meaningful — without efficient gradient-based training, activation functions would be academic curiosities. Hinton has noted that the success of ReLU surprised the community because the field had long assumed smooth, differentiable activations everywhere were necessary for training to work.

researcher · Vinod Nair | b. ~1980 | ReLU in Deep Learning

Vinod Nair, working with Geoffrey Hinton at the University of Toronto, co-authored the landmark 2010 paper "Rectified Linear Units Improve Restricted Boltzmann Machines" that brought ReLU to the attention of the deep learning community. Before this work, the neural network field largely assumed that activation functions needed to be smooth and differentiable everywhere — Nair and Hinton's empirical results challenged this assumption decisively. The surprising historical footnote is that ReLU had appeared in earlier neuroscience-inspired models of neural firing rates, but was largely ignored by the machine learning mainstream until this paper.

Figure
ReLU Variants — Shape Comparison
x f(x) -3 -1.5 0 1.5 3 0 1 -1 ReLU Leaky ReLU ELU GELU Activation Function Shapes
Figure
Dead Neuron Problem — Gradient Flow
Gradient Flow Through ReLU ACTIVE (x > 0) x=2 ReLU 2 grad=1.0 grad=1.0 DEAD (x < 0) x=-2 ReLU 0 grad=0 BLOCKED grad=0 ✕

Intuition

The Mental Model

Imagine a bouncer at a nightclub whose only job is to decide whether information gets through or not. For any positive signal, the bouncer waves it through unchanged — what comes in is exactly what goes out. For any negative signal, the bouncer blocks it completely, outputting zero. That's ReLU. Now imagine why this bouncer is surprisingly good at their job: they create a natural competition among neurons. At any given moment, roughly half your network is "switched on" (the neurons whose weighted input happens to be positive for this particular data point), and they handle the task. The other half are quietly resting. This sparsity — most neurons inactive at any moment — turns out to make representations cleaner and training faster. Leaky ReLU is like a bouncer who lets whispering through for blocked signals, just in case it's useful. GELU is a probabilistic bouncer who uses a smooth probability curve instead of a hard cutoff — which is why Transformer models prefer it.

Historical Development

Origins · 1 minhow this was discovered
History
1943
McCulloch & Pitts introduce the first mathematical neuron model, using a hard threshold (step function) — the conceptual ancestor of all activation functions.
1958
Frank Rosenblatt's Perceptron uses a binary threshold activation, showing neurons can learn simple patterns but failing for anything nonlinear.
1986
Rumelhart, Hinton, and Williams popularize backpropagation with sigmoid activations, enabling multi-layer networks — but sigmoid's small gradients begin causing vanishing gradient problems in deeper stacks.
2000
Hahnloser et al. publish "Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit," using a rectification operation inspired by how biological neurons fire only for positive signals — an early ML appearance of ReLU.
2010
Vinod Nair and Geoffrey Hinton publish "Rectified Linear Units Improve Restricted Boltzmann Machines," establishing ReLU as a serious alternative to sigmoid in the deep learning context.
2012
AlexNet (Krizhevsky, Sutskever, Hinton) wins ImageNet by a massive margin, using ReLU throughout — this is the moment ReLU becomes the default activation for the field.
2015
Clevert et al. introduce ELU (Exponential Linear Unit), offering smoother behavior for negative inputs and mean activations closer to zero, improving training on some tasks.
2016
Maas et al.'s Leaky ReLU and He et al.'s PReLU (Parametric ReLU, where α is learned) are consolidated as practical fixes for the dying neuron problem.
2018
Hendrycks and Gimpel propose GELU (Gaussian Error Linear Unit), which becomes the activation of choice for Transformer-based models including BERT and all GPT variants.
2020
Ramachandran et al.'s Swish activation (x · sigmoid(x)) and subsequent automated activation search methods demonstrate that neural architecture search can discover novel activations — but ReLU and GELU remain dominant in practice.

Why It Matters

Real-World Impact
AlexNet (2012)The paper that reignited deep learning uses ReLU in all five convolutional layers and three fully-connected layers. The authors explicitly report that ReLU training was 6× faster than an equivalent tanh network — without this speedup, training on the hardware of 2012 would have been impractical.
ResNet (Microsoft, 2015)All 152 layers of ResNet use ReLU activations paired with batch normalization. The combination allows gradients to flow through an unprecedented network depth; ResNet architectures still underpin most image recognition systems deployed today.
GPT-2 / GPT-3 / GPT-4 (OpenAI)The feedforward sublayers inside every Transformer block use GELU (a smooth ReLU variant). In GPT-3's 96-layer architecture, this means GELU is applied billions of times per forward pass — making its cheap compute cost critical at scale.
BERT (Google, 2018)Uses GELU throughout its 12 or 24 Transformer layers. Google's production search ranking systems are derived from BERT variants, meaning ReLU-family activations directly influence billions of search queries every day.
YOLO Object DetectionThe real-time object detection family (used in autonomous vehicles, surveillance, and robotics) relies on Leaky ReLU in its early versions to prevent dead neurons in the convolutional backbone, where many feature maps can have negative activations.
Stable Diffusion / Diffusion ModelsThe U-Net backbone used in image generation models like Stable Diffusion uses SiLU (Sigmoid Linear Unit, also called Swish) — another ReLU variant — in its residual blocks, chosen for its smooth gradient properties during the iterative denoising process.
On-device ML (phones, edge devices)Standard ReLU is the activation of choice for MobileNet and EfficientNet families deployed on smartphones, because it requires zero multiplication for negative inputs — saving precious compute cycles and battery life on constrained hardware.

Prerequisites

What This Leads To

See Also

Sigmoid Activation Function · Batch Normalization · Vanishing and Exploding Gradients · Loss Functions · Feedforward Neural Networks

Before this

Where this leads