BERT: Bidirectional Encoder Representations

BERT (Bidirectional Encoder Representations from Transformers), introduced by Devlin et al. at Google in 2018, is a pre-trained language model that learns deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in every layer. It matters because it established the dominant "pre-train then fine-tune" paradigm that underpins nearly every state-of-the-art NLP system in production today. The counterintuitive fact: BERT is never trained to predict the next word — instead it learns language by randomly masking tokens and predicting them from surrounding context, a seemingly simple objective that produces richer representations than any directional language model before it.

The core insight

Before BERT, language models read text in only one direction — left-to-right or right-to-left — which means every word representation is "blind" to the words that come after it. BERT's breakthrough is making every layer of the Transformer encoder attend to the entire sequence simultaneously, so the representation of the word "bank" in "river bank" is shaped by both "river" (to the left) and whatever comes after (to the right) at every depth of the network. This bidirectionality is achieved not by architecture wizardry but by a training objective called Masked Language Modeling: randomly hide 15% of tokens and ask the model to reconstruct them from full context. The "aha moment" is realising that a single pre-trained BERT checkpoint can be fine-tuned with just one extra linear layer to achieve state-of-the-art performance on tasks as different as question answering, sentiment analysis, and named entity recognition.

How It Works

Input Representation

BERT takes a token sequence and constructs its input embedding by summing three learned embedding tables:

1
ei=Exitok+Esiseg+Eipos\quad \mathbf{e}_i = \mathbf{E}^{\text{tok}}_{x_i} + \mathbf{E}^{\text{seg}}_{s_i} + \mathbf{E}^{\text{pos}}_i

Explanation: The final embedding for position i is the sum of a token embedding, a segment embedding (sentence A vs. B), and a positional embedding — all three are learned jointly during pre-training.

  • Etok∈RV×d\mathbf{E}^{\text{tok}} \in \mathbb{R}^{V \times d}: token embedding matrix, VV = vocabulary size (30,522 for BERT-base), dd = 768
  • Eseg∈R2×d\mathbf{E}^{\text{seg}} \in \mathbb{R}^{2 \times d}: two embeddings distinguishing sentence A from sentence B in paired-sentence tasks
  • Epos∈R512×d\mathbf{E}^{\text{pos}} \in \mathbb{R}^{512 \times d}: learned absolute positional embeddings (not sinusoidal as in the original Transformer)

Scaled Dot-Product Self-Attention

Within each of BERT's 12 (base) or 24 (large) Transformer encoder layers, the scaled dot-product attention mechanism computes:

2
Attention(Q,K,V)=softmax ⁣(QK⊤dk)V\quad \text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d_k}}\right)\mathbf{V}

Explanation: Each query vector attends to all key vectors, the dot-product scores are scaled to prevent vanishing gradients in the softmax, and the resulting weights form a convex combination of value vectors.

  • Q=XWQ\mathbf{Q} = \mathbf{X}\mathbf{W}^Q, K=XWK\mathbf{K} = \mathbf{X}\mathbf{W}^K, V=XWV\mathbf{V} = \mathbf{X}\mathbf{W}^V — projections of the same input X∈Rn×d\mathbf{X} \in \mathbb{R}^{n \times d}
  • dk=d/hd_k = d / h where hh is the number of heads (12 for BERT-base)
  • The dk\sqrt{d_k} scaling prevents dot products from growing large in magnitude as dimension increases

Multi-Head Attention

BERT runs hh attention heads in parallel, then concatenates and projects:

3
MHA(X)=Concat(head1,…,headh) WO\quad \text{MHA}(\mathbf{X}) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)\,\mathbf{W}^O

Explanation: Running multiple heads lets the model simultaneously attend to different types of relationships — syntactic, semantic, coreference — at different subspace positions.

  • Each headi=Attention(XWiQ, XWiK, XWiV)\text{head}_i = \text{Attention}(\mathbf{X}\mathbf{W}^Q_i,\ \mathbf{X}\mathbf{W}^K_i,\ \mathbf{X}\mathbf{W}^V_i)
  • WO∈Rd×d\mathbf{W}^O \in \mathbb{R}^{d \times d}: output projection matrix
  • For BERT-base: h=12h=12, d=768d=768, dk=dv=64d_k = d_v = 64; for BERT-large: h=16h=16, d=1024d=1024

Masked Language Modeling Objective

The MLM loss is a standard cross-entropy over only the masked positions:

4
LMLM=−1∣M∣∑i∈Mlog⁡P(xi∣x~)\quad \mathcal{L}_{\text{MLM}} = -\frac{1}{|\mathcal{M}|}\sum_{i \in \mathcal{M}} \log P(x_i \mid \tilde{\mathbf{x}})

Explanation: The model is penalised only on the masked tokens; non-masked positions contribute no gradient signal, which is why BERT sees bidirectional context without "cheating" by looking at the token it must predict.

  • M\mathcal{M}: the set of masked token indices (≈15% of tokens per sequence)
  • x~\tilde{\mathbf{x}}: the corrupted input (masked tokens replaced with `[MASK]`, a random token, or left unchanged per the 80/10/10 rule)
  • P(xi∣x~)P(x_i \mid \tilde{\mathbf{x}}): softmax over the full vocabulary from the final hidden state at position ii

Next Sentence Prediction Objective

BERT's second pre-training task is a binary classification on the `[CLS]` token:

5
LNSP=−[ylog⁡P(IsNext)+(1−y)log⁡P(NotNext)]\quad \mathcal{L}_{\text{NSP}} = -\left[y \log P(\text{IsNext}) + (1-y)\log P(\text{NotNext})\right]

Explanation: The model learns whether sentence B naturally follows sentence A in the original document, forcing the [CLS] embedding to capture inter-sentence coherence — though later work showed NSP adds marginal benefit compared to MLM alone.

  • y∈{0,1}y \in \{0,1\}: label, 1 if the pair is consecutive, 0 if sentence B is a random negative
  • P(IsNext)P(\text{IsNext}): sigmoid applied to a single linear projection of the `[CLS]` final hidden state

---

PyTorch Fine-Tuning Skeleton

```python from transformers import BertTokenizer, BertForSequenceClassification import torch tokenizer = BertTokenizer.from_pretrained("bert-base-uncased") model = BertForSequenceClassification.from_pretrained( "bert-base-uncased", num_labels=2)

# Tokenise a batch enc = tokenizer( ["The river bank was flooded.", "I deposited cash at the bank."], padding=True, truncation=True, max_length=128, return_tensors="pt" )

# Forward pass — the [CLS] hidden state is pooled + projected internally outputs = model(**enc, labels=torch.tensor([0, 1])) loss, logits = outputs.loss, outputs.logits loss.backward() # only the task head + BERT weights fine-tune together ``` The `[CLS]` token at position 0 accumulates a sentence-level representation through all 12 layers of self-attention; the classification head is simply a linear layer Wc∈Rd×nlabels\mathbf{W}_c \in \mathbb{R}^{d \times n_{\text{labels}}} applied to that vector. ---

researcher · Jacob Devlin | b. ~1985 | Lead Author of BERT

Jacob Devlin was a research scientist at Google AI Language when he led the team that published BERT in October 2018, building directly on the Transformer architecture of Vaswani et al. His key insight was that making the pre-training objective bidirectional — rather than the left-to-right approach used by OpenAI's GPT — could be achieved cleanly through masked token prediction, yielding dramatically stronger representations without architectural changes. A surprising fact: the original BERT paper was posted to arXiv just two months before it swept the GLUE benchmark leaderboard by large margins, and Devlin has said the team was initially uncertain whether the approach would outperform ELMo and GPT by such a wide margin.

researcher · Ashish Vaswani | b. ~1984 | Transformer Architecture

Ashish Vaswani was a senior researcher at Google Brain when he co-authored "Attention Is All You Need" (2017), the paper that introduced the Transformer architecture upon which BERT is directly built — BERT uses the Transformer's encoder stack unchanged except for learned positional embeddings. His contribution to this topic is foundational: without the parallelisable, fully-attentive encoder block, the bidirectional pre-training that defines BERT would be computationally intractable at scale. Vaswani later co-founded Adept AI, and he has noted that the team originally expected the Transformer to be most valuable for machine translation, not as a general-purpose representation learning engine.

Figure
BERT Architecture and Pre-Training Objectives
BERT: Bidirectional Encoder Stack + Pre-Training Objectives Input: [CLS] river [MASK] was flooded [SEP] near town [SEP] Transformer Encoder Stack Layer 12 — Multi-Head Self-Attention (all positions attend to all positions) → FFN Layer 2 — Multi-Head Self-Attention → FFN Layer 1 — Multi-Head Self-Attention → FFN ← bidirectional context → Output: C T₁ T₂ T₃ T₄ T₅ T₆ T₇ T₈ NSP head MLM head Classification (C) Token Task (T₁..Tₙ) Span Extraction (T_s..T_e) Masked token Special token Regular token BERT-base: 12 layers, 768d, 12 heads, 110M params
Figure
Masked Language Modeling Token Selection
MLM: The 80 / 10 / 10 Masking Rule Original: flooded the village near the river ← selected (15%) 80% → [MASK] token replaced with [MASK] 10% → random replaced with a random vocab token 10% → unchanged original token kept; model must still predict Loss computed only on selected position → model uses full bidirectional context to reconstruct 10%+10% prevent model from assuming [MASK] only ever appears at pre-training time (train/test mismatch)

Intuition

The Mental Model

Imagine you are doing a crossword puzzle where you must fill in a blank word given all the clues — clues from letters to the left and letters to the right of the blank, clues from intersecting words above and below. You are not forced to read only left-to-right; you use the full two-dimensional context of the grid. That is exactly what BERT does for every masked word in a sentence. Contrast this with how a child might be taught to read: "sound it out left to right." That child, upon encountering "I went to the river ___" has to predict "bank" before seeing the next sentence about depositing money. BERT instead sees both sentences simultaneously and can use "depositing money" in the next sentence to inform its representation of "bank" in the current one — something no left-to-right language model can do. The pre-training / fine-tuning split has a vivid analogy too: think of BERT's pre-training as a student spending years reading every Wikipedia article and every book, building rich background knowledge. Fine-tuning is then a one-week crash course for a specific exam — the student needs very few labelled examples because their background knowledge is already so deep. The single linear layer added on top of BERT for classification is that crash course: just enough to redirect pre-existing knowledge toward a new task. Finally, the `[CLS]` token is a designated "notepad" that the model learns to write a sentence-level summary onto across 12 layers of bidirectional attention — by the time you read off the `[CLS]` hidden state, it has aggregated information from every other token in both sentences.

Historical Development

Origins · 1 minhow this was discovered
History
2013
Mikolov et al. introduce Word2Vec — static, context-free word embeddings; "bank" always maps to the same vector regardless of meaning.
2015
Peters et al. begin work on TagLM, showing that language model representations improve NLP taggers — a precursor to contextualised embeddings.
2017
Vaswani et al. publish "Attention Is All You Need" — the Transformer architecture that BERT's encoder is built on verbatim.
2018
(Feb): Howard & Ruder release ULMFiT — the first demonstration that fine-tuning a pre-trained language model outperforms task-specific architectures across many NLP benchmarks.
2018
(Jun): Radford et al. at OpenAI release GPT-1 — a left-to-right Transformer pre-trained on BooksCorpus, achieving strong results via fine-tuning but limited by unidirectionality.
2018
(Oct): Devlin, Chang, Lee & Toutanova at Google release BERT on arXiv — sets new state-of-the-art on 11 NLP benchmarks including SQuAD 1.1, MNLI, and GLUE, often by large margins.
2019
Liu et al. release RoBERTa — trains BERT longer on more data with larger batches, removes NSP, finds that BERT was significantly undertrained; RoBERTa substantially improves on all BERT benchmarks.
2019
Lan et al. introduce ALBERT — reduces BERT's parameters via cross-layer weight sharing and factorised embeddings, enabling larger models with less memory.
2020
Clark et al. release ELECTRA — replaces MLM with a "replaced token detection" discriminator task, achieving