Long Short-Term Memory (LSTM)

Long Short-Term Memory (LSTM) is a recurrent neural network architecture introduced by Hochreiter and Schmidhuber in 1997 that uses a gated cell state mechanism to selectively retain or discard information across arbitrarily long sequences, directly solving the vanishing gradient problem that crippled earlier RNNs. LSTMs remain foundational to sequence modeling in speech recognition, machine translation, and time-series forecasting, even as Transformers dominate many benchmarks. Remarkably, the LSTM's forget gate — now considered essential — was not in the original 1997 paper but was added by Gers et al. in 1999, meaning the architecture most practitioners use today is already a revision of the "original."

The core insight

A standard RNN collapses all past information into a single hidden state vector, forcing every new input to overwrite the past — like writing on a whiteboard you must erase before adding anything new. The LSTM's breakthrough is maintaining a separate cell state (the "memory tape") that flows through time with only lightweight, multiplicative interactions: three learned gates decide what to forget, what to write, and what to read at each step. This means gradients can flow backward through time along the cell state highway with near-zero attenuation, making it possible to learn dependencies spanning hundreds of timesteps. The "aha moment": gating is just sigmoid-weighted element-wise multiplication — beautiful in its simplicity.

How It Works

An LSTM processes a sequence one timestep at a time. At step $t$ it receives input $x_t \in \mathbb{R}^d$ and the previous state tuple $(h_{t-1}, c_{t-1})$ , then produces updated states $(h_t, c_t)$ . Every gate is a learned sigmoid applied to a linear combination of $x_t$ and $h_{t-1}$ .

Forget gate — decides what fraction of the old cell state to keep:

\quad f_t = \sigma(W_f \, x_t + U_f \, h_{t-1} + b_f)

Explanation: Each element of $f_t \in (0,1)$ independently gates how much of the corresponding cell-state dimension survives; 0 means "completely forget," 1 means "remember perfectly."

$W_f \in \mathbb{R}^{H \times d}$ , $U_f \in \mathbb{R}^{H \times H}$ — learned weight matrices
$b_f \in \mathbb{R}^H$ — bias (often initialized to 1 to encourage remembering at the start of training)
$\sigma$ — element-wise sigmoid

Input gate and candidate cell — decides what new information to write:

\quad i_t = \sigma(W_i \, x_t + U_i \, h_{t-1} + b_i), \qquad \tilde{c}_t = \tanh(W_c \, x_t + U_c \, h_{t-1} + b_c)

Explanation: $i_t$ modulates how strongly the candidate values $\tilde{c}_t \in (-1,1)^H$ are written into memory — the gate and candidate together act as a controlled write head.

Cell state update — the memory highway:

\quad c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t

Explanation: The new cell state is an additive blend of the (gated) old memory and the (gated) new candidate, so gradients can flow backward through time via the $f_t \odot c_{t-1}$ term with no repeated matrix multiplication.

$\odot$ — element-wise (Hadamard) product

Output gate and hidden state — decides what to expose:

\quad o_t = \sigma(W_o \, x_t + U_o \, h_{t-1} + b_o), \qquad h_t = o_t \odot \tanh(c_t)

Explanation: The hidden state $h_t$ is a tanh-compressed readout of the cell state, further filtered by $o_t$ , so the network can hold information in $c_t$ without necessarily broadcasting it downstream.

Gradient flow insight — the key reason LSTMs work is that the gradient of the loss with respect to an earlier cell state $c_k$ passes through the chain:

\quad \frac{\partial c_t}{\partial c_k} = \prod_{\tau=k+1}^{t} f_\tau

Explanation: Because this product involves only the forget gates — bounded scalars, not full Jacobian matrices — the gradient neither explodes nor vanishes as dramatically as in vanilla RNNs, provided the forget gates stay near 1.

PyTorch Implementation

```python import torch import torch.nn as nn

# Single-layer LSTM: input_size=128, hidden_size=256 lstm = nn.LSTM(input_size=128, hidden_size=256, num_layers=1, batch_first=True)

# x: (batch=32, seq_len=100, input_size=128) x = torch.randn(32, 100, 128)

# Initialize hidden and cell states h0 = torch.zeros(1, 32, 256) # (num_layers, batch, hidden) c0 = torch.zeros(1, 32, 256)

# Forward pass — output: (batch, seq_len, hidden) output, (hn, cn) = lstm(x, (h0, c0)) print(output.shape) # torch.Size([32, 100, 256]) print(hn.shape) # torch.Size([1, 32, 256]) print(cn.shape) # torch.Size([1, 32, 256])

# For stateful inference (e.g., streaming):

# detach states to truncate BPTT graph h0, c0 = hn.detach(), cn.detach() ``` ```python

# Manual LSTM cell (educational — shows every gate) class LSTMCell(nn.Module): def __init__(self, input_size, hidden_size): super().__init__() self.hidden_size = hidden_size

# Pack all four gate weight matrices into one for efficiency self.W = nn.Linear(input_size + hidden_size, 4 * hidden_size) def forward(self, x, state): h, c = state combined = torch.cat([x, h], dim=-1) # (batch, d + H) gates = self.W(combined) # (batch, 4H)

# Split into four equal chunks f, i, g, o = gates.chunk(4, dim=-1) f = torch.sigmoid(f) # forget gate i = torch.sigmoid(i) # input gate g = torch.tanh(g) # candidate o = torch.sigmoid(o) # output gate c_new = f c + i g h_new = o * torch.tanh(c_new) return h_new, c_new ```

researcher · Sepp Hochreiter | b. 1967 | Invented LSTM and Diagnosed the Vanishing Gradient Problem

Hochreiter, an Austrian computer scientist, identified the vanishing gradient problem rigorously in his 1991 diploma thesis — years before the community broadly recognized it — and co-invented LSTM with Jürgen Schmidhuber as a direct solution. His 1997 paper with Schmidhuber introduced the constant error carousel (the cell state), arguably the most consequential architectural idea in sequence modeling before the Transformer. A lesser-known fact: Hochreiter's diploma thesis advisor initially downplayed the vanishing gradient analysis, and the work went largely unnoticed for several years until the LSTM paper brought it to prominence.

researcher · Jürgen Schmidhuber | b. 1963 | Co-invented LSTM and Championed Credit Assignment in Deep Networks

Schmidhuber, a Swiss-German AI researcher at IDSIA in Lugano, co-authored the foundational 1997 LSTM paper and has spent decades working on meta-learning, neural program synthesis, and credit assignment across time. His group produced several LSTM extensions including peephole connections (Gers & Schmidhuber, 2000) and Connectionist Temporal Classification (CTC), which enabled the first practical end-to-end speech recognizers. Schmidhuber is famously outspoken about attribution in AI history, maintaining a detailed timeline of discoveries he argues the field has misattributed.

Figure

LSTM Cell — Gate Data Flow

Figure

Vanishing Gradient: RNN vs LSTM

Intuition

The Mental Model

Imagine you are a court stenographer transcribing a long legal proceeding. You have two tools: a notepad (the cell state $c_t$ ) and a speaking voice (the hidden state $h_t$ ). The notepad can hold important facts — a witness's name, a key date — across hours of testimony without degradation, because you simply underline things rather than rewriting them. Your voice, however, is selective: you only read aloud from the notepad when it's relevant to the current question. The three gates map naturally to this metaphor: the forget gate is you crossing out old facts that are no longer relevant ("the defendant's alibi has been disproven, strike it"); the input gate and candidate are you deciding whether and how strongly to write a new fact into the notepad; and the output gate is you deciding whether to mention what's on the notepad right now or stay quiet. The magic is that facts on the notepad survive unchanged until explicitly crossed out, so a name written at timestep 1 can be read at timestep 500 with no distortion — exactly what the additive cell-state update achieves mathematically.

Historical Development

Origins · 1 minhow this was discovered

History

1991

Sepp Hochreiter's diploma thesis (TU Munich) provides the first rigorous mathematical analysis of the vanishing and exploding gradient problems in RNNs — largely ignored for several years.

1994

Yoshua Bengio, Patrice Simard, and Paolo Frasconi publish "Learning Long-Term Dependencies with Gradient Descent is Difficult," independently formalizing the vanishing gradient barrier and casting doubt on whether standard RNNs could ever solve long-range problems.

1997

Hochreiter and Schmidhuber publish "Long Short-Term Memory" in Neural Computation, introducing the cell state, forget-less input/output gates, and the constant error carousel concept. The paper is initially met with skepticism.

1999

Felix Gers, Jürgen Schmidhuber, and Fred Cummins publish "Learning to Forget: Continual Prediction with LSTM," adding the forget gate — the element most practitioners today consider essential — fixing catastrophic interference on continual-prediction tasks.

2000

Gers and Schmidhuber introduce peephole connections, allowing gates to inspect the cell state directly, improving performance on precise timing tasks.

2005

Alex Graves and Schmidhuber combine LSTM with Connectionist Temporal Classification (CTC), enabling the first sequence-to-sequence labeling without pre-segmented data, a breakthrough for handwriting recognition.

2013

Alex Graves demonstrates deep bidirectional LSTM with CTC achieving near-human performance on offline handwriting recognition, marking LSTM's entry into production-grade systems.

2014

Kyunghyun Cho et al. propose the Gated Recurrent Unit (GRU), a streamlined two-gate variant that matches LSTM on many benchmarks with fewer parameters, spawning debate about which architecture to prefer.

2014–2016

Google deploys LSTM-based models for Google Voice Search, Gmail Smart Reply, and Google Translate, bringing LSTM to billions of users and cementing it as the dominant production sequence model.

2017

Vaswani et al. publish "Attention Is All You Need," introducing the Transformer and beginning LSTM's gradual displacement on NLP benchmarks — though LSTMs retain advantages in streaming, low-latency, and on-device settings.

Why It Matters

Real-World Impact

Google Voice Search (2015)Google replaced their GMM-HMM pipeline with a deep LSTM acoustic model, reducing word error rate by ~30% and becoming the first large-scale production deployment of LSTMs for speech recognition serving hundreds of millions of queries daily.

Gmail Smart Reply (2016)Google's Smart Reply system used a stacked LSTM encoder-decoder to generate short contextual email responses, processing billions of emails; the LSTM encoder distilled entire email threads into a single hidden state used to condition reply generation.

Apple Siri & QuickType (2016–2017)Apple integrated bidirectional LSTMs into Siri's on-device speech recognition and the QuickType keyboard's next-word prediction, prioritizing LSTMs over Transformers due to their O(1) inference time per token — critical for real-time mobile use.

Google Translate's Neural MT (2016)Google's production Neural Machine Translation system used an 8-layer stacked LSTM encoder-decoder with attention, replacing a decade-old phrase-based system and improving BLEU scores by 55–85% on major language pairs overnight.

DeepMind AlphaFold 1 (2018)The original AlphaFold used bidirectional LSTMs over multiple sequence alignments to predict inter-residue distances, contributing to protein structure prediction; later superseded by attention mechanisms in AlphaFold 2.

Financial Time-Series ForecastingHedge funds and quantitative trading firms use stacked LSTMs for multi-step volatility forecasting and order-book modeling, where the ability to carry state across variable-length market regimes without fixed context windows is operationally advantageous over Transformers.

Clinical NLP and EHR ModelingLSTMs are widely deployed for modeling patient timelines in electronic health records, where irregular sampling intervals and the need for online (streaming) inference without reprocessing full histories make them preferable to Transformer-based alternatives in latency-constrained hospital systems.

Long Short-Term Memory (LSTM)

How It Works

PyTorch Implementation

Intuition

Historical Development

Why It Matters

Prerequisites

What This Leads To

See Also

Before this

Where this leads