Masked Language Modeling

Masked Language Modeling (MLM) is a self-supervised pre-training technique where a model learns to predict randomly hidden ("masked") words in a sentence using only the surrounding context. It matters enormously in modern AI because it allows transformers like BERT to learn rich, bidirectional language understanding from raw text — no human labels required. The counterintuitive fact: a model trained to do nothing but fill in blanks turns out to be one of the most powerful general-purpose language understanding systems ever built.

The core insight

The "aha moment" is this: by forcing a model to reconstruct missing words, you inadvertently teach it everything about language — grammar, facts, relationships, even common sense — because the only way to reliably fill in a blank is to genuinely understand the surrounding context. It's a beautiful trick: the training task is trivially defined (mask some tokens, predict them back), but the knowledge required to solve it is vast. Unlike reading a sentence left-to-right, MLM lets the model look in both directions at once, making its understanding richer than traditional language models.

How It Works

MLM training proceeds in three clean steps: mask, encode, predict.

Step 1 — Masking. Given a sequence of tokens, randomly select 15% of them. Of those selected tokens:

  • 80% are replaced with a special `[MASK]` token
  • 10% are replaced with a random other token
  • 10% are left unchanged

This mixed strategy (not just always masking) prevents the model from learning to only pay attention to `[MASK]` symbols. It must treat every token as potentially important.

Step 2 — Encoding. The corrupted sequence is passed through a transformer encoder, which produces a contextual embedding for every token position — masked or not — simultaneously.

Step 3 — Predicting. For each masked position, the model uses the embedding at that position to predict what the original token was, via a linear layer followed by a softmax over the entire vocabulary. The loss function is:

1
LMLM=−∑i∈Mlog⁡P(xi∣x~)\quad \mathcal{L}_{\text{MLM}} = -\sum_{i \in \mathcal{M}} \log P(x_i \mid \tilde{x})

Explanation: The model is penalized for assigning low probability to the true original token at each masked position — the better its guesses, the smaller the loss. Where:

  • M\mathcal{M} is the set of masked token positions
  • xix_i is the true original token at position ii
  • x~\tilde{x} is the corrupted input sequence (with masks)
  • P(xi∣x~)P(x_i \mid \tilde{x}) is the model's predicted probability for the correct token

The masking probability is typically fixed:

2
pmask=0.15\quad p_{\text{mask}} = 0.15

Explanation: Only 15% of tokens are selected for prediction — enough signal to learn from, but enough context left visible that the task is solvable. Here is a minimal Python sketch of the masking procedure: ```python import random MASK_TOKEN = "[MASK]" VOCAB = ["the", "cat", "sat", "on", "mat", "dog", "ran", "a"] def apply_mlm_mask(tokens, mask_prob=0.15): labels = [] # what we want the model to predict masked = list(tokens) for i, token in enumerate(tokens): if random.random() < mask_prob: labels.append(token) # save the true label roll = random.random() if roll < 0.80: masked[i] = MASK_TOKEN # 80%: replace with [MASK] elif roll < 0.90: masked[i] = random.choice(VOCAB) # 10%: random token

# else: 10% leave unchanged, label still recorded else: labels.append(None) # not masked, no prediction needed return masked, labels tokens = ["the", "cat", "sat", "on", "the", "mat"] masked_tokens, labels = apply_mlm_mask(tokens) print("Original:", tokens) print("Masked: ", masked_tokens) print("Labels: ", labels) ``` Example output: ``` Original: ['the', 'cat', 'sat', 'on', 'the', 'mat'] Masked: ['the', '[MASK]', 'sat', 'on', '[MASK]', 'mat'] Labels: [None, 'cat', None, None, 'the', None] ``` The model only computes loss on positions where `labels[i] is not None`.

researcher · Jacob Devlin | b. 1983 | Lead author of BERT

Jacob Devlin is a research scientist at Google who led the team that introduced BERT (Bidirectional Encoder Representations from Transformers) in 2018, the paper that made MLM the dominant pre-training paradigm. His key insight was combining the transformer encoder architecture with MLM and a second task (Next Sentence Prediction) to produce embeddings that shattered benchmarks across 11 NLP tasks simultaneously. A surprising fact: BERT's name was partly chosen as a nod to the Sesame Street character, starting a tradition of NLP models named after Muppets (ELMO, ERNIE, Big Bird).

Figure
Masked Language Modeling — Data Flow
Masked Language Modeling — Forward Pass INPUT the [MASK] sat on [MASK] mat Transformer Encoder (bidirectional self-attention over ALL tokens) OUTPUT emb₁ emb₂★ emb₃ emb₄ emb₅★ emb₆ → predict "cat" → predict "the" masked visible
Figure
MLM vs Causal LM — Attention Pattern Comparison
Attention Patterns: MLM vs Causal LM MLM (Bidirectional) the [M] sat on the [M] sat on Every token sees all other tokens ✓ Causal LM (Left-only) the cat sat on the cat sat on Each token sees only past tokens →

Intuition

The Mental Model

Imagine you're doing a crossword puzzle. To fill in a missing word, you don't just read the clue — you look at intersecting words above, below, left, and right, all at once. That's exactly what MLM does. It removes words from a sentence and asks the model: "given everything around this blank, what was here?" Compare this to a different game where you read a story one word at a time, always guessing the next word — that's causal language modeling. The crossword version (MLM) forces much richer reasoning because you must reconcile clues from both sides of the gap. After billions of such puzzles on web text, the model has effectively memorized the rules of the whole crossword grid — which is language itself. Fine-tuning later is just telling the model which specific squares of that giant grid your application cares about.

Historical Development

Origins · 1 minhow this was discovered
History
1953
Claude Shannon introduced the concept of a "cloze" task — deleting words from text and asking humans to fill them in — as a way to measure text difficulty and information content.
2013
Word2Vec (Mikolov et al.) demonstrated that predicting masked context words (the CBOW variant) could produce surprisingly meaningful word embeddings, planting the seed for MLM-style training.
2017
Vaswani et al. published "Attention Is All You Need," introducing the transformer architecture that would become MLM's engine, based entirely on self-attention without recurrence.
2018
Peter Howard and Sebastian Ruder released ELMo and ULMFiT respectively, showing that pre-training a language model on raw text and then fine-tuning on a downstream task was a powerful general strategy.
2018
(Oct): Jacob Devlin et al. at Google published "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," introducing MLM as we know it today. BERT immediately achieved state-of-the-art on 11 NLP benchmarks.
2019
RoBERTa (Liu et al., Facebook AI) showed that BERT was significantly undertrained, removing the Next Sentence Prediction task and training on more data with dynamic masking, improving performance substantially.
2019
SpanBERT (Joshi et al.) replaced single-token masking with contiguous span masking, producing stronger representations for span-prediction tasks like question answering.
2020
ELECTRA (Clark et al.) replaced the standard MLM decoder with a "replaced token detection" discriminator, training 4× more efficiently while matching BERT's performance on a fraction of the compute.
2022–2023
MLM principles extended beyond text — masked autoencoders (MAE, He et al.) applied the same masking-and-reconstruction idea to images and video, proving MLM is a general self-supervised learning principle.

Why It Matters

Real-World Impact
BERT (Google, 2018)The original MLM model. Used in Google Search from 2019 onward to better understand natural language queries — Google stated it affected 10% of all English searches immediately on launch, one of the biggest leaps in Search's history.
BioBERT / SciBERTDomain-specific BERT variants pre-trained with MLM on PubMed biomedical abstracts and scientific papers respectively. Used by pharmaceutical companies and research institutions to extract drug-gene interactions and accelerate literature review.
CodeBERT (Microsoft)MLM applied to paired code and natural language, enabling tasks like code search (finding functions from English descriptions) and code documentation generation, integrated into developer tooling.
Sentence-BERTBuilt on MLM pre-training, adds a fine-tuning step to produce sentence-level embeddings. Used in semantic search engines, duplicate question detection, and customer support ticket clustering at scale.
HuggingFace Transformers LibraryProvides off-the-shelf MLM models (BERT, RoBERTa, DistilBERT, etc.) that millions of developers use via three lines of Python, making MLM-based NLP the default starting point for any text classification or NER task.
Clinical NLP (NHS, Epic Systems)MLM models fine-tuned on de-identified clinical notes power named entity recognition pipelines that extract diagnoses, medications, and procedures from free-text clinical records automatically.
Multilingual BERT (mBERT)A single MLM model trained on 104 languages simultaneously. Remarkably, it transfers knowledge across languages — fine-tune on English data and it partially works in French or Swahili, enabling NLP in low-resource languages.

Prerequisites

What This Leads To

See Also

BERT and Its Variants · Causal Language Modeling · Self-Attention Mechanism · Transfer Learning in NLP · Tokenization

Before this

Where this leads