Contextual Embeddings and ELMo

A static embedding maps each word type $w$ in a fixed vocabulary to a single vector $\mathbf{e}_w \in \mathbb{R}^d$, learned once from a corpus and then frozen. This is the representation produced by word2vec, GloVe, or fastText. Its core limitation is lexical ambiguity collapse: polysemous words (bank, bat, spring), words whose sense shifts with register or domain, and words entangled in idioms all get squashed into one averaged vector that must serve every context simultaneously. A contextual embedding instead defines a function

that produces the representation of token $w_i$ as a function of the entire input sequence, not just $w_i$ in isolation. The same word type can now receive different vectors depending on its neighbors. ELMo instantiates $f$ as the internal hidden states of a deep bidirectional LSTM language model, so representations come "for free" as a byproduct of a well-understood, well-studied pretraining objective: predicting the next word (and the previous word) in a sentence. Two design choices make ELMo work well in practice. First, the model is bidirectional but not naively so: it trains a genuinely separate forward LSTM (reading left-to-right, predicting $w_{t+1}$ from $w_1,\ldots,w_t$) and a genuinely separate backward LSTM (reading right-to-left, predicting $w_{t-1}$ from $w_n,\ldots,w_{t+1}$), then concatenates their states. This avoids the training-time information leak that would occur if a single recurrent pass could see the target word through the other direction (an issue that motivated masked, rather than autoregressive, objectives in later architectures). Second, ELMo does not simply expose the top LSTM layer as "the" embedding. Instead, it exposes all layers — the character-CNN input layer and every biLSTM layer — and lets each downstream task learn its own softmax-normalized weighted sum. This is justified empirically and theoretically: different layers of a deep LSTM language model specialize in different levels of linguistic structure, so a syntax-heavy task (like part-of-speech tagging) and a semantics-heavy task (like word-sense disambiguation or question answering) should draw on different layers with different emphasis.

Step 1 — Character-level input representation. Rather than a lookup table indexed by word identity (which requires a closed vocabulary and fails on out-of-vocabulary words), ELMo represents each word by running its character sequence through a convolutional layer, followed by a highway network and a linear projection. This produces a context-independent word vector $\mathbf{x}_k$ for word $w_k$, built purely from its spelling, before any recurrence happens. This is important: it means unknown words, misspellings, and morphologically rich languages are all handled gracefully, since the model never needs an explicit "UNK" fallback.

Step 2 — Forward and backward language models. Given the sequence of context-independent vectors $\mathbf{x}_1, \ldots, \mathbf{x}_n$, a forward LSTM language model computes

by running an $L$-layer stacked LSTM forward through the sequence, producing hidden states $\overrightarrow{\mathbf{h}}_{k,j}$ at layer $j \in \{1,\ldots,L\}$ and position $k$. Symmetrically, a backward LSTM language model computes

producing hidden states $\overleftarrow{\mathbf{h}}_{k,j}$. The two directions share the character-CNN input layer and the output softmax vocabulary projection, but maintain separate LSTM parameters, so the forward LSTM never has access to right-context and vice versa. Training jointly maximizes the sum of the forward and backward log-likelihoods.

Step 3 — Assembling per-layer representations. For each token $k$ and each layer $j = 1, \ldots, L$, concatenate the two directions:

Together with the layer-0 "token representation" $\mathbf{h}_{k,0} = [\mathbf{x}_k ; \mathbf{x}_k]$ (the character-CNN output, duplicated to match dimensionality), this gives $L+1$ candidate vectors per token.

Step 4 — Task-specific layer mixing. For a specific downstream task $\textit{task}$, ELMo computes

where $s^{\textit{task}} = \text{softmax}(\mathbf{a}^{\textit{task}})$ are learned, task-specific softmax-normalized layer weights, and $\gamma^{\textit{task}}$ is a learned scalar that rescales the entire vector (useful because the distribution of the biLM's internal states can differ substantially from what the downstream task's optimizer expects). Both $\mathbf{a}^{\textit{task}}$ and $\gamma^{\textit{task}}$ are trained on the downstream task while the biLM's own weights are typically kept frozen.

Step 5 — Downstream integration. The resulting $\text{ELMo}_k$ vector is concatenated with (or substituted for) the static embedding for $w_k$ and fed into the downstream task's architecture (e.g., a BiLSTM tagger, a span-extraction QA model), which is trained normally with backpropagation, updating everything except the frozen biLM weights. ```

# ----- Pretraining phase ----- function TRAIN_BILM(corpus, L, vocab): initialize CharCNN, Highway, forward_LSTM (L layers), backward_LSTM (L layers), softmax_out for each sentence (w_1, ..., w_n) in corpus:

# context-independent token vectors for k in 1..n: x_k = Highway(CharCNN(chars(w_k)))

# forward pass: left-to-right, causal fwd_h[0] = zero_state for k in 1..n: fwd_h[k] = forward_LSTM(x_k, fwd_h[k-1]) # all L layers p_fwd_k = softmax_out(fwd_h[k][L]) loss += -log p_fwd_k[w_{k+1}] # predict NEXT word

# backward pass: right-to-left, causal in reverse bwd_h[n+1] = zero_state for k in n..1: bwd_h[k] = backward_LSTM(x_k, bwd_h[k+1]) p_bwd_k = softmax_out(bwd_h[k][L]) loss += -log p_bwd_k[w_{k-1}] # predict PREVIOUS word backpropagate(loss) update(CharCNN, Highway, forward_LSTM, backward_LSTM, softmax_out) return CharCNN, Highway, forward_LSTM, backward_LSTM

# ----- Feature extraction / fine-tuning phase ----- function ELMO_EMBED(sentence, biLM, s_task, gamma_task): x = [Highway(CharCNN(chars(w))) for w in sentence] fwd_h = run_forward(biLM.forward_LSTM, x) # L layers, per position bwd_h = run_backward(biLM.backward_LSTM, x) # L layers, per position for k in 1..len(sentence): h[k][0] = concat(x[k], x[k]) # layer-0 = token repr, doubled for j in 1..L: h[k][j] = concat(fwd_h[k][j], bwd_h[k][j]) elmo = [] for k in 1..len(sentence): weighted_sum = sum(s_task[j] h[k][j] for j in 0..L) elmo.append(gamma_task weighted_sum) return elmo # one contextual vector per token

function TRAIN_DOWNSTREAM(task_data, biLM, task_model): freeze(biLM) # biLM weights fixed initialize s_task (softmax weights), gamma_task, task_model for (sentence, label) in task_data: elmo_vecs = ELMO_EMBED(sentence, biLM, s_task, gamma_task) input_reprs = [concat(static_embed(w), elmo_vecs[k]) for k, w in enumerate(sentence)] prediction = task_model(input_reprs) loss = task_loss(prediction, label) backpropagate(loss) # updates s_task, gamma_task, update(s_task, gamma_task, task_model) # and task_model only ```

Time complexity. Let $n$ be sentence length, $L$ the number of stacked LSTM layers, $d$ the hidden dimension, and $|V|$ the vocabulary size for the softmax output. Character-CNN encoding of a word of length $c$ costs $O(c \cdot d)$ per word (convolution + highway), so $O(n \cdot c \cdot d)$ total, typically negligible compared to the recurrence. Each LSTM layer processes the sequence with $O(n)$ sequential steps, each step costing $O(d^2)$ (matrix multiplications for gates), giving $O(n \cdot d^2)$ per layer and $O(L \cdot n \cdot d^2)$ for the full stack, doubled for the two directions: $O(L \cdot n \cdot d^2)$ overall (constant factor 2 absorbed). The softmax output layer costs $O(n \cdot |V| \cdot d)$ per direction unless a sampled or hierarchical softmax is used, in which case this drops to $O(n \cdot d \cdot \log|V|)$ or $O(n \cdot d \cdot k)$ for $k$ negative samples. Crucially, the LSTM recurrence is inherently sequential: step $t$'s hidden state depends on step $t-1$'s, so within one direction the $n$ steps cannot be parallelized across time, even though the two directions (forward/backward) and different sentences in a batch can run in parallel. This is the central efficiency contrast with attention-based architectures, where all positions can be processed simultaneously.

Space complexity. Storing all hidden states across all $L$ layers and both directions (needed for the downstream layer-mixing step) requires $O(L \cdot n \cdot d)$ memory per sentence. Training additionally requires storing activations for backpropagation through time, which is also $O(L \cdot n \cdot d)$, since BPTT must retain every intermediate gate activation to compute gradients.

Correctness of the bidirectional objective (no information leak). A natural but incorrect design would be to run a single bidirectional LSTM whose forward and backward hidden states are concatenated at every layer and then used to jointly predict the current word $w_k$. This is invalid, because concatenating $\overrightarrow{\mathbf{h}}_{k}$ (which has seen $w_1,\ldots,w_k$) with $\overleftarrow{\mathbf{h}}_{k}$ (which has seen $w_k,\ldots,w_n$) and then feeding that combined state through further shared layers before predicting $w_k$ would let the backward path leak information about $w_k$'s left context into a representation used to predict $w_k$ itself — worse, if layers are shared and re-processed jointly, the network could shortcut and effectively "copy" $w_k$ from the backward path, driving training loss to near-zero without learning anything useful. ELMo avoids this by keeping the forward and backward LSTMs entirely separate stacks of parameters trained on two separate autoregressive objectives:

and only after both models are trained are their hidden states concatenated for use as a feature (never fed back into either LM's own prediction). The invariant maintained throughout training is: the forward LSTM's hidden state at position $k$ is a function only of $w_1, \ldots, w_{k-1}$, and the backward LSTM's hidden state at position $k$ is a function only of $w_{k+1}, \ldots, w_n$. This invariant is preserved by construction (each recurrence only ever consumes strictly-past, resp. strictly-future, tokens) and it is exactly what makes maximizing $\mathcal{L}$ a valid proxy for learning the true joint distribution of language, rather than a degenerate shortcut.

Why layer-mixing is justified. Each LSTM layer's hidden state is trained (implicitly, via backpropagation from the language-modeling loss) to be useful for predicting the next/previous word using only what the previous layer computed. Layers do not receive an explicit supervision signal telling them "encode syntax" or "encode semantics" — but empirically (via probing classifiers on ELMo's layers, reported in the original work) lower layers achieve stronger performance on tasks that are more local and syntactic (e.g., part-of-speech tagging), while higher layers do better on tasks requiring longer-range semantic disambiguation (e.g., word-sense disambiguation). Because the ideal mixture is task-dependent and not knowable in advance, learning a softmax-normalized convex combination $s_j^{\textit{task}}$ is a principled way to let each downstream task extract exactly the blend of abstraction levels it needs, while the shared biLM backbone stays frozen and reusable — this is the same idea that later motivates layer-wise probing and layer-selection strategies in Transformer architectures.

Tradeoffs. ELMo's sequential recurrence caps training throughput and cannot exploit the full parallelism available in modern GPU/TPU hardware the way self-attention can, which is one reason Transformer-based models (relying on the Attention Mechanism in NLP rather than recurrence) eventually replaced biLSTM language models as the default pretraining architecture. ELMo also freezes the pretrained biLM and only adapts a lightweight mixing layer, which is cheap and avoids catastrophic forgetting, but it sacrifices the deeper task-specific adaptation that full fine-tuning (as in BERT) can achieve.

Take the sentence: "The bank raised interest rates." and contrast it with "They sat by the river bank." We trace how ELMo would represent the token "bank" differently in each.

The essential contrast: a static embedding gives $\mathbf{e}_{\text{bank}}^{A} = \mathbf{e}_{\text{bank}}^{B}$ by definition (cosine similarity exactly 1, since it's the same lookup vector), while ELMo gives two vectors whose cosine similarity is noticeably lower, reflecting genuinely different senses — this is the direct, measurable payoff of context-dependence, and it is exactly what downstream disambiguation-sensitive tasks (word-sense disambiguation, coreference resolution, semantic role labeling) benefit from.

ELMo's release in 2018 triggered an immediate, measurable jump in state-of-the-art results across a wide range of NLP benchmarks — question answering, textual entailment, semantic role labeling, coreference resolution, named-entity recognition, and sentiment analysis — simply by swapping static embeddings for ELMo features in existing task architectures, with no other architectural change. This demonstrated conclusively that large-scale unsupervised pretraining, transferred via contextual representations, was a more powerful lever than continued hand-engineering of task-specific architectures. That lesson directly shaped the design of Transformer architectures such as BERT, GPT, and their successors, all of which retain ELMo's core recipe — pretrain broadly on unlabeled text, transfer to downstream tasks — while replacing the recurrent backbone with the parallelizable Attention Mechanism in NLP and, in BERT's case, replacing autoregressive language modeling with a masked-token objective that lets a single stack see both directions safely. Contextual embeddings from ELMo and its successors are used throughout industrial NLP systems: search engines rely on contextual query and document representations for relevance ranking, virtual assistants use them for intent classification and slot filling, machine translation systems use contextual encoders, and content-moderation and sentiment-analysis pipelines rely on them to disambiguate sarcasm, negation, and domain-specific word senses that static embeddings cannot capture. Understanding ELMo also clarifies, by contrast, exactly what problem attention-based pretraining was designed to solve — sequential bottlenecks and one-directional shortcuts — making it a natural conceptual stepping stone toward modern large language models.

Ask the tutor: How does ELMo's forward/backward separation compare to BERT's masked-language-model approach to achieving safe bidirectionality?