Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are a class of neural networks designed to process sequential data by maintaining a hidden state that is updated at each time step, allowing information to persist across the sequence. They are foundational to modern sequence modeling in NLP, speech, and time-series analysis, enabling models to condition outputs on arbitrarily long input histories. A counterintuitive fact: despite being designed to capture long-range dependencies, vanilla RNNs are theoretically proven to forget information exponentially fast as sequence length grows — the very problem they were invented to solve.

The core insight

The "aha moment" for RNNs is that a neural network can be made to process sequences by feeding the same set of weights repeatedly across time steps, while passing a compressed summary — the hidden state — from one step to the next. Think of it as a single neural network being unrolled into a chain, where each copy shares weights but receives new input and the memory of everything that came before. This weight sharing is what makes the model generalize across time positions, just as convolutional weight sharing generalizes across spatial positions. The hidden state is the network's working memory: it is the only bridge between past and future.

Common trapCommon Misconception

Many practitioners assume that because an RNN's hidden state can in principle carry information from arbitrarily far back, it actually does so in practice. This is wrong. The hidden state is updated by a squashing nonlinearity (typically tanh) at every step, which compresses and mixes all past information into a fixed-size vector; gradients flowing back through dozens of tanh applications either vanish toward zero or explode toward infinity. The correct mental model is that a trained vanilla RNN has an effective memory window of roughly 5–20 steps for typical initializations — anything beyond that requires architectural interventions like LSTMs, GRUs, or attention mechanisms. Do not conflate "has access to all past steps in principle" with "actually uses information from all past steps in practice."

How It Works

At each time step $t$ , an RNN takes an input vector $x_t$ and the previous hidden state $h_{t-1}$ and produces a new hidden state $h_t$ . All time steps share the same weight matrices.

\quad h_t = \tanh(W_{hh}\, h_{t-1} + W_{xh}\, x_t + b_h)

Explanation: The new hidden state is a nonlinear compression of two linear signals — one carrying memory from the past hidden state, one carrying the current input.

Symbols:

$h_t \in \mathbb{R}^d$ — hidden state vector at step $t$ , dimension $d$ is a hyperparameter
$W_{hh} \in \mathbb{R}^{d \times d}$ — recurrent weight matrix (past → present)
$W_{xh} \in \mathbb{R}^{d \times n}$ — input weight matrix; $n$ is input dimension
$b_h \in \mathbb{R}^d$ — bias vector
$\tanh$ — element-wise hyperbolic tangent, squashing values to $(-1, 1)$

The output at each step (e.g., for sequence-to-sequence or language modeling) is:

\quad \hat{y}_t = \text{softmax}(W_{hy}\, h_t + b_y)

Explanation: The hidden state is linearly projected and normalized into a probability distribution over output classes at each time step.

Symbols:

$W_{hy} \in \mathbb{R}^{|\mathcal{V}| \times d}$ — output weight matrix; $|\mathcal{V}|$ is vocabulary or class size
$b_y \in \mathbb{R}^{|\mathcal{V}|}$ — output bias

The loss over an entire sequence of length $T$ is summed cross-entropy:

\quad \mathcal{L} = -\sum_{t=1}^{T} \log \hat{y}_t[y_t]

Explanation: The total sequence loss is the sum of the negative log-probability assigned to the correct target at every time step.

Symbols:

$y_t$ — ground-truth class index at step $t$
$\hat{y}_t[y_t]$ — the model's predicted probability for the correct class

Training uses Backpropagation Through Time (BPTT), which unrolls the computation graph across all $T$ steps and applies the chain rule. The gradient of the loss with respect to the hidden state at an earlier step $k$ involves a product of Jacobians:

\quad \frac{\partial \mathcal{L}}{\partial h_k} = \frac{\partial \mathcal{L}}{\partial h_T} \prod_{t=k+1}^{T} \frac{\partial h_t}{\partial h_{t-1}} = \frac{\partial \mathcal{L}}{\partial h_T} \prod_{t=k+1}^{T} W_{hh}^\top \cdot \text{diag}\!\left(1 - h_t^2\right)

Explanation: The gradient must be multiplied through every time step between $k$ and $T$ , so it is scaled by the same recurrent weight matrix repeated $(T-k)$ times, causing exponential decay or explosion.

Symbols:

$\text{diag}(1 - h_t^2)$ — diagonal Jacobian of tanh; entries are $\tanh'(z) = 1 - \tanh^2(z) \in (0,1]$
The product of matrices each with spectral radius $< 1$ (typical after tanh) causes vanishing gradients

A minimal PyTorch implementation illustrates the core loop: ```python import torch import torch.nn as nn class VanillaRNN(nn.Module): def __init__(self, input_size: int, hidden_size: int, output_size: int): super().__init__() self.hidden_size = hidden_size

# Fused input+hidden linear for efficiency self.rnn_cell = nn.RNNCell(input_size, hidden_size, nonlinearity='tanh') self.output_layer = nn.Linear(hidden_size, output_size) def forward(self, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:

# x: (seq_len, batch, input_size) seq_len, batch, _ = x.shape h = torch.zeros(batch, self.hidden_size, device=x.device) # h_0 outputs = [] for t in range(seq_len): h = self.rnn_cell(x[t], h) # Eq.[1] applied here outputs.append(self.output_layer(h)) # Eq.[2] applied here return torch.stack(outputs, dim=0), h # (seq_len, batch, output_size), final h ```

researcher · Sepp Hochreiter | b. 1967 | Vanishing Gradient Problem & LSTM

Sepp Hochreiter, a German computer scientist, identified and rigorously characterized the vanishing and exploding gradient problem in recurrent networks in his 1991 diploma thesis — years before it was widely recognized. Together with Jürgen Schmidhuber, he proposed the Long Short-Term Memory (LSTM) architecture in 1997, which introduced gating mechanisms specifically designed to let gradients flow over long sequences. Remarkably, Hochreiter's 1991 analysis was written in German and remained largely unknown to the English-speaking community for nearly a decade.

researcher · Yoshua Bengio | b. 1964 | Theoretical Analysis of Gradient Flow in RNNs

Yoshua Bengio, a Canadian AI researcher and 2018 Turing Award winner, co-authored the landmark 1994 paper "Learning Long-Term Dependencies with Gradient Descent is Difficult," providing the first rigorous mathematical proof that gradient-based methods struggle with long-range dependencies in RNNs. His work shifted the community's understanding from treating training difficulty as a practical nuisance to a fundamental mathematical barrier. Bengio is also known for his foundational contributions to word embeddings and deep learning theory, and he co-founded Mila — the Quebec AI Institute.

Figure

Unrolled RNN Architecture and Hidden State Flow

Figure

Vanishing Gradient: Signal Decay Over Time Steps

Intuition

The Mental Model

Imagine you are reading a novel, but you can only keep a small sticky note of reminders as you turn each page. At every new page, you read the page's content and update your sticky note — erasing some old notes, adding new ones. At the end, everything you know about the story is compressed into whatever fits on that sticky note. That sticky note is the RNN's hidden state. The process of reading each page and rewriting the sticky note is exactly Equation [1]. The crucial insight is that the same sticky-note-update rules are applied at every page: the RNN doesn't have different "reading strategies" for page 1 versus page 200 — it reuses the same weights everywhere. This is weight sharing across time. Now imagine the catch: your sticky note can only hold 10 bullet points, and every time you update it, old bullets tend to get smudged and overwritten. By the time you reach page 200, almost nothing from page 1 survives legibly. This is vanishing gradients — the model loses the ability to learn from distant past context. LSTMs fix this by giving you a laminated card alongside your sticky note, one where you can deliberately choose what to protect and what to erase using gates.

Historical Development

Origins · 1 minhow this was discovered

History

1986

Rumelhart, Hinton & Williams publish "Learning Representations by Back-Propagating Errors," which, while focused on feedforward nets, lays out the general backpropagation framework that BPTT later extends to sequences.

1990

Elman introduces "Elman Networks" in "Finding Structure in Time" — simple RNNs with a context layer fed from the previous hidden state, directly inspiring modern architectures.

1991

Sepp Hochreiter's diploma thesis provides the first rigorous analysis of the vanishing and exploding gradient problem in deep and recurrent networks, though it circulates mostly in German academic circles.

1994

Bengio, Simard & Frasconi publish "Learning Long-Term Dependencies with Gradient Descent is Difficult," formally proving the exponential decay of gradient signals over time in RNNs.

1997

Hochreiter & Schmidhuber publish "Long Short-Term Memory" in Neural Computation, introducing the LSTM with input, forget, and output gates — the dominant sequence model for the next two decades.

2000

Gers, Schmidhuber & Cummins add the forget gate to the original LSTM, producing the architecture that most modern implementations use.

2014

Cho et al. introduce the Gated Recurrent Unit (GRU) in "Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation," a simpler alternative to LSTM with competitive performance.

2015

Karpathy's "The Unreasonable Effectiveness of Recurrent Neural Networks" blog post demonstrates that character-level RNNs trained on raw text learn surprisingly rich structure, popularizing RNNs among practitioners.

2017

Vaswani et al. publish "Attention Is All You Need," introducing the Transformer; within two years, self-attention largely replaces RNNs for long-sequence NLP tasks due to parallelism and better long-range modeling.

Why It Matters

Real-World Impact

Google Translate (pre-2017)Used stacked LSTM encoder–decoder architectures with attention for neural machine translation, replacing phrase-based statistical methods and achieving dramatic BLEU score improvements across language pairs.

Apple Siri & Amazon Alexa (speech recognition)Deployed deep bidirectional LSTMs for acoustic modeling in ASR pipelines, where the RNN processes spectral features frame-by-frame to produce phoneme probabilities in real time.

DeepMind's WaveNet (2016)Used dilated causal convolutional RNNs to synthesize raw audio waveforms at 16kHz, producing the first neural text-to-speech system with naturalness comparable to human speech.

OpenAI's GPT-1 (2018)While GPT-1 used Transformers, its predecessor language models (e.g., AWD-LSTM by Merity et al., 2017) used weight-dropped LSTMs to achieve state-of-the-art perplexity on Penn Treebank and WikiText-103.

Financial time-series modelingLSTM-based models are deployed in high-frequency trading systems to predict short-term price movements from order-book sequences, where the recurrent inductive bias over fixed-length windows outperforms feedforward baselines.

Handwriting generation (Graves, 2013)An LSTM mixture-density network conditioned on text produces realistic, stylistically varied handwriting by modeling pen trajectories as Gaussian mixtures — a landmark demonstration of RNN generative power.

Music generation (Google Magenta)Melody RNN uses LSTM networks to generate musically coherent note sequences conditioned on prior notes and rhythmic grid positions, deployed in real creative tools for musicians.

Recurrent Neural Networks

How It Works

Intuition

Historical Development

Why It Matters

Prerequisites

What This Leads To

See Also

Before this

Where this leads