ELMo: Contextual Word Embeddings

ELMo (Embeddings from Language Models) is a deep contextualized word representation technique introduced by Peters et al. (2018) that computes word embeddings as a learned linear combination of internal states from a bidirectional LSTM language model, producing representations that vary based on surrounding context rather than assigning a single static vector per word. It matters because ELMo was the first widely adopted method to demonstrate that rich, task-agnostic pre-trained language representations could dramatically improve performance across nearly every NLP benchmark simultaneously. The counterintuitive fact: ELMo does not replace your task model's embedding layer — it augments it, and the optimal weighting of its internal layers differs per task, meaning syntax and semantics are actually stored at different depths.

The core insight

Every prior word embedding method (Word2Vec, GloVe) assigns one fixed vector to each word regardless of context — "bank" gets the same representation whether you're discussing finance or rivers. ELMo's breakthrough is that a word's representation is computed on the fly from the hidden states of every layer of a bidirectional language model, so the same surface form produces fundamentally different vectors in different sentences. The model is pre-trained once on a massive corpus, and then the internal representations are exported as features for any downstream task. The "aha moment" is realizing that the internal layers specialize: lower layers capture syntax, upper layers capture semantics, and the task-specific weighting learns which information it needs.

Common trapCommon Misconception

The most common mistake is treating ELMo as a drop-in replacement that simply produces "better word vectors" in the same way you'd swap GloVe for Word2Vec. What practitioners actually get wrong is believing ELMo produces a single embedding per word — it produces L+1 vectors per token (one per BiLSTM layer plus the character CNN output), and the final representation is a learned, task-specific weighted sum of all of them. Using only the top layer (which feels natural coming from RNN hidden-state intuition) consistently underperforms the full combination. The correct mental model is that ELMo is a feature extractor that exposes a stack of representations at different levels of linguistic abstraction, and the downstream task's scalar mix weights are trained parameters that decide how to blend those levels.

How It Works

ELMo builds on a two-stage pipeline: first, a character-level CNN converts each token's characters into an initial token representation; second, a stacked bidirectional LSTM processes the full sequence, and all internal states are linearly combined with learned weights to produce the final embedding.

Stage 1 — Character CNN Encoding Each word is encoded character-by-character using convolutional filters followed by highway networks, producing a context-independent representation $\mathbf{x}_k^{LM}$ for token $k$ . This sidesteps out-of-vocabulary problems entirely.

Stage 2 — Bidirectional Language Model The forward LM predicts each token given its left context:

\quad \sum_{k=1}^{N} \log p\!\left(t_k \mid t_1, \dots, t_{k-1};\, \Theta_x, \overrightarrow{\Theta}_{LSTM}, \Theta_s\right)

Explanation: This is the log-likelihood objective for the forward pass — maximize the probability of each token given all preceding tokens, summed over the full sequence of length N.

$t_k$ : the token at position $k$
$\Theta_x$ : parameters of the character CNN token encoder
$\overrightarrow{\Theta}_{LSTM}$ : parameters of the forward LSTM stack
$\Theta_s$ : parameters of the softmax output layer

The backward LM runs the same computation in reverse:

\quad \sum_{k=1}^{N} \log p\!\left(t_k \mid t_{k+1}, \dots, t_N;\, \Theta_x, \overleftarrow{\Theta}_{LSTM}, \Theta_s\right)

Explanation: The backward pass predicts each token from its right context, giving the model knowledge of what comes after — the two objectives are jointly optimized. The combined BiLM objective simply maximizes the sum of both directions, tying the token encoder $\Theta_x$ and softmax $\Theta_s$ parameters but keeping LSTM parameters separate:

\quad \mathcal{L} = \sum_{k=1}^{N}\left[\log p\!\left(t_k \mid t_{1:k-1}\right) + \log p\!\left(t_k \mid t_{k+1:N}\right)\right]

Explanation: Jointly optimizing forward and backward language modeling forces the LSTM to build representations that are useful for predicting context in both directions simultaneously.

Stage 3 — ELMo Representation For a model with $L$ BiLSTM layers, token $k$ has $2L+1$ raw representations: the character CNN output plus the forward and backward hidden states at each layer. These are collapsed into a single task-specific vector:

\quad \mathbf{ELMo}_k^{task} = \gamma^{task} \sum_{j=0}^{L} s_j^{task}\, \mathbf{h}_{k,j}^{LM}

Explanation: The final ELMo vector is a scalar-weighted sum of all layer representations, scaled by a task-specific amplitude γ — both the weights s and the scale γ are learned during fine-tuning on the downstream task.

$\gamma^{task}$ : scalar that scales the entire ELMo vector (learned per task)
$s_j^{task}$ : softmax-normalized scalar weight for layer $j$ (learned per task, $\sum_j s_j = 1$ )
$\mathbf{h}_{k,j}^{LM}$ : concatenation of forward and backward hidden states at layer $j$ for token $k$
$L$ : number of BiLSTM layers (2 in the original ELMo paper)

In practice the ELMo vector is concatenated with the task model's existing word embeddings rather than replacing them: ```python import torch import torch.nn as nn from allennlp.modules.elmo import Elmo, batch_to_ids

# Load pre-trained ELMo options_file = "elmo_2x4096_512_2048cnn_2xhighway_options.json" weight_file = "elmo_2x4096_512_2048cnn_2xhighway_weights.hdf5" elmo = Elmo(options_file, weight_file, num_output_representations=1, # one task-specific mix dropout=0.5) sentences = [["The", "bank", "is", "steep"], ["The", "bank", "approved", "the", "loan"]] character_ids = batch_to_ids(sentences) # shape: (B, T, 50) embeddings = elmo(character_ids)

# embeddings['elmo_representations'][0]: (B, T, 1024)

# "bank" will have different 1024-d vectors in the two sentences ```

researcher · Matthew Peters | b. 1980 | Inventor of ELMo

Matthew Peters was a research scientist at the Allen Institute for AI (AI2) with a background in physics before transitioning to NLP, making him emblematic of the interdisciplinary nature of modern deep learning research. His 2018 paper "Deep Contextualized Word Representations" (ELMo) demonstrated for the first time that a single pre-trained language model could simultaneously set new state-of-the-art results on six diverse NLP benchmarks including NER, SRL, coreference resolution, and sentiment analysis. Surprisingly, Peters had published a predecessor paper in 2017 using only the top LSTM layer as context embeddings, and the leap to using all layers — the crucial insight — came from an ablation study done almost as an afterthought.

Figure

ELMo Architecture: Character CNN → Stacked BiLSTM → Scalar Mix

Figure

Scalar Mix Weights Vary by Task

Intuition

The Mental Model

Imagine you're a linguistics expert who reads a sentence and must label every word's meaning. You don't decide what "bank" means until you've read the whole sentence — you hold all possible meanings in your head simultaneously and resolve them from context. ELMo is like having three specialists examine each word at different levels of linguistic detail: the first specialist (the character CNN) knows how the word is spelled and can guess its part of speech even if they've never seen it before. The second specialist (LSTM layer 1) understands grammar — they see how the word fits into the sentence's syntactic skeleton. The third specialist (LSTM layer 2) understands meaning in context — they know that "bank" in a sentence about loans must be a financial institution. When you (the downstream task) ask for ELMo's opinion, you ask all three specialists, but you assign different trust weights to each depending on what you care about: a named-entity recognizer trusts the grammar expert most, while a semantic role labeler trusts the meaning expert most. Crucially, all three specialists were trained together by asking them to predict the next and previous words in billions of sentences — a surprisingly powerful self-supervised signal that forces them to build rich internal world models.

Historical Development

Origins · 1 minhow this was discovered

History

2013

Mikolov et al. introduce Word2Vec — static word embeddings learned from co-occurrence, one vector per word type regardless of context.

2014

Pennington et al. introduce GloVe, training on global co-occurrence statistics; still fully context-independent, "bank" is one point in vector space.

2015

Kiros et al. propose Skip-Thought Vectors, learning sentence-level representations from a seq2seq prediction objective — an early hint that LM objectives could produce transferable features.

2017

McCann et al. publish CoVe (Contextualized Word Vectors), using MT encoder hidden states as context-aware embeddings; proves contextualization helps but is limited by the MT training signal.

2017

Peters et al. (AI2) release a predecessor using only the top BiLSTM layer as "context2vec"-style embeddings, showing gains on NER but not yet exploiting multi-layer representations.

2018

Peters et al. publish "Deep Contextualized Word Representations" (ELMo), introducing the scalar mix across all BiLSTM layers; sets state-of-the-art on 6 NLP benchmarks in one paper — a shock to the community.

2018

Howard & Ruder publish ULMFiT, demonstrating that fine-tuning the entire language model (rather than freezing it) improves results further; ELMo and ULMFiT together define the "pre-train then adapt" paradigm.

2018

Devlin et al. introduce BERT, replacing the BiLSTM with a Transformer and using masked LM instead of causal LM — directly inheriting ELMo's insight about bidirectionality and multi-layer representations but scaling massively.

2019

Tenney et al. publish "BERT Rediscovers the Classical NLP Pipeline," confirming ELMo's observation that lower layers encode syntax and upper layers encode semantics, now in Transformers — validating ELMo's architectural intuitions at scale.

Why It Matters

Real-World Impact

Named Entity Recognition (NER)ELMo's release immediately set a new state-of-the-art on CoNLL 2003 NER (F1: 92.22%), a benchmark that had been incrementally improved for years — demonstrating that richer representations could substitute for extensive feature engineering.

Semantic Role Labeling (SRL)Peters et al. showed a 3.9 F1 absolute improvement over the prior best system on OntoNotes SRL by simply adding ELMo embeddings; the improvement came entirely from the upper LSTM layers, confirming the layer-specialization hypothesis.

Coreference ResolutionAllenNLP's coreference system using ELMo achieved 70.4 F1 on OntoNotes, a gain of over 3 points — tasks that require long-range semantic understanding benefited most from the backward pass giving every token knowledge of future context.

Question Answering (SQuAD)ELMo added to the BiDAF model improved EM score from 68.0 to 72.1 on SQuAD 1.1, showing that even sophisticated QA architectures with their own attention mechanisms benefited from pre-trained contextual features.

BERT and Subsequent TransformersBERT explicitly cites ELMo as its primary motivation, adopting the bidirectionality and multi-layer representation ideas while scaling to transformers; every modern LLM (GPT-4, Gemini, Claude) inherits the pre-train-then-adapt paradigm that ELMo validated at scale.

spaCy and Production NLP PipelinesspaCy v2 integrated ELMo-style contextual embeddings into its default pipeline for NER and dependency parsing, bringing contextualized representations into production NLP systems at scale for the first time.

Scientific NLP (SciBERT, BioBERT)ELMo inspired domain-specific pre-training; Allen AI's SciBERT directly extended ELMo's approach to scientific text, demonstrating that domain-specific pre-training of language models improves performance on technical benchmarks by 5–10 points.

ELMo: Contextual Word Embeddings

How It Works

Intuition

Historical Development

Why It Matters

Prerequisites

What This Leads To

See Also

Before this