The Decoder-Only Transformer

The decoder-only transformer is a neural network architecture that autoregressively generates sequences by attending only to previously produced tokens, forming the backbone of virtually every large language model in production today including GPT-4, Claude, Llama, and Gemini. It matters because its simplicity relative to the full encoder-decoder transformer — dropping cross-attention entirely — turns out to lose almost nothing in practice while dramatically simplifying training at scale. The counterintuitive fact: despite being called "decoder-only," the architecture learns rich bidirectional-like representations of its context window through the key-value cache, achieving encoder-quality understanding without a separate encoder.

The core insight

A decoder-only transformer is just a stack of masked self-attention layers: every token can see all tokens before it, but none after it. This causal masking means the model can be trained on raw text with a single next-token prediction objective — the labels are the inputs shifted by one position, so no manual annotation is ever needed. The "aha moment" is that language modeling, the simplest possible self-supervised task, forces the network to build an internal world model rich enough to generate coherent reasoning, code, and prose — no special architecture required, just scale and causal masking.

How It Works

Input Embedding and Positional Encoding

Raw tokens (integers from a vocabulary of size $V$ ) are first mapped to dense vectors, then augmented with positional information. Modern decoder-only models use learned absolute or rotary positional encodings rather than the sinusoidal encodings from the original transformer.

\quad \mathbf{x}_i = \mathbf{W}_E[\text{token}_i] + \mathbf{p}_i

Explanation: The input representation for position $i$ is the sum of a token embedding (row $\text{token}_i$ of the embedding matrix $\mathbf{W}_E \in \mathbb{R}^{V \times d}$ ) and a positional vector $\mathbf{p}_i \in \mathbb{R}^d$ .

Symbols: $\mathbf{W}_E$ is the token embedding matrix with $V$ rows (one per vocabulary item) and $d$ columns (model dimension); $\text{token}_i$ is the integer token id at position $i$ ; $\mathbf{p}_i$ is the positional encoding at position $i$ .

Causal (Masked) Self-Attention

The heart of the architecture is scaled dot-product attention with a causal mask. For a sequence of length $T$ with model dimension $d$ and head dimension $d_k = d/H$ across $H$ heads:

\quad \text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^\top + \mathbf{M}}{\sqrt{d_k}}\right)\mathbf{V}

Explanation: Each query vector attends to all key vectors, scaled to prevent vanishing gradients, then the additive mask $\mathbf{M}$ sets future positions to $-\infty$ so they contribute zero weight after softmax.

Symbols: $\mathbf{Q}, \mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times d_k}$ are the query, key, and value matrices for one head; $d_k$ is the per-head dimension; $\mathbf{M} \in \mathbb{R}^{T \times T}$ is the causal mask with $M_{ij} = 0$ if $j \le i$ else $-\infty$ . The projections that produce Q, K, V are:

\quad \mathbf{Q} = \mathbf{X}\mathbf{W}_Q, \quad \mathbf{K} = \mathbf{X}\mathbf{W}_K, \quad \mathbf{V} = \mathbf{X}\mathbf{W}_V

Explanation: Each of the three matrices $\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V \in \mathbb{R}^{d \times d_k}$ is a learned linear projection that maps the full-dimensional token representations $\mathbf{X}$ into a lower-dimensional query, key, or value space for one attention head. Multi-head attention concatenates $H$ heads and projects back:

\quad \text{MHA}(\mathbf{X}) = \text{Concat}(\text{head}_1, \ldots, \text{head}_H)\,\mathbf{W}_O

Explanation: The outputs of all $H$ attention heads (each $\in \mathbb{R}^{T \times d_k}$ ) are concatenated into a $\mathbb{R}^{T \times d}$ matrix and projected by $\mathbf{W}_O \in \mathbb{R}^{d \times d}$ to produce the final attention output.

Feed-Forward Sublayer

After attention, each position is transformed independently by a two-layer MLP (the feed-forward network, FFN). In modern models this is typically a SwiGLU or GeGLU variant, but the canonical form is:

\quad \text{FFN}(\mathbf{x}) = \max(0,\, \mathbf{x}\mathbf{W}_1 + \mathbf{b}_1)\,\mathbf{W}_2 + \mathbf{b}_2

Explanation: Each token's representation $\mathbf{x} \in \mathbb{R}^d$ is expanded to an intermediate dimension $d_\text{ff}$ (typically $4d$ ) by $\mathbf{W}_1$ , passed through ReLU nonlinearity, then projected back to $d$ by $\mathbf{W}_2$ , adding representational capacity that attention alone lacks.

Residual Connections and Layer Norm

Each sublayer (attention and FFN) is wrapped in a Pre-LayerNorm residual connection (the modern convention, differing from the original "post-norm" paper): ```python

# Pre-LN decoder block in PyTorch class DecoderBlock(nn.Module): def __init__(self, d_model, n_heads, d_ff, dropout=0.1): super().__init__() self.ln1 = nn.LayerNorm(d_model) self.ln2 = nn.LayerNorm(d_model) self.attn = CausalSelfAttention(d_model, n_heads) self.ffn = nn.Sequential( nn.Linear(d_model, d_ff), nn.GELU(), nn.Linear(d_ff, d_model), nn.Dropout(dropout), ) def forward(self, x, mask=None): x = x + self.attn(self.ln1(x), mask) # residual + attention x = x + self.ffn(self.ln2(x)) # residual + FFN return x ```

Causal Mask Construction

```python import torch def causal_mask(seq_len: int, device: torch.device) -> torch.Tensor: """Upper-triangular matrix of -inf, zeros on and below diagonal.""" mask = torch.full((seq_len, seq_len), float('-inf'), device=device) mask = torch.triu(mask, diagonal=1) # keep diagonal (self-attention OK) return mask # shape: (T, T) ```

Training Objective

The model is trained by maximum likelihood on next-token prediction across all positions simultaneously — the key efficiency advantage over autoregressive inference: ```python

# Cross-entropy loss for language modeling logits = model(input_ids) # (B, T, V) loss = F.cross_entropy( logits[:, :-1, :].reshape(-1, vocab_size), input_ids[:, 1:].reshape(-1), # targets: shifted by one ) ```

researcher · Ashish Vaswani | b. 1983 | Inventor of the Transformer

Vaswani led the Google Brain team that published "Attention Is All You Need" (2017), introducing the full encoder-decoder transformer, which the decoder-only architecture directly descends from. His specific contribution was demonstrating that self-attention alone — without recurrence or convolution — could match or exceed the state of the art in sequence modeling. Vaswani later co-founded Essential AI, and the original transformer paper remains one of the most cited in the history of machine learning.

researcher · Alec Radford | b. 1990 | GPT and the Decoder-Only Scaling Paradigm

Radford, as a research scientist at OpenAI, was the lead author on the GPT-1 paper (2018) which first demonstrated that a decoder-only transformer pretrained on raw text with next-token prediction could be fine-tuned to achieve state-of-the-art results across diverse NLP tasks. He followed this with GPT-2 (2019) and GPT-3 (co-authored, 2020), establishing the scaling laws that now govern LLM development. Radford also co-invented CLIP, showing that the same transformer ideas extend to vision-language alignment.

Figure

Decoder-Only Transformer Architecture

Figure

KV-Cache: Efficient Autoregressive Inference

Intuition

The Mental Model

Imagine you are an author writing a novel one word at a time, but you have a perfect photographic memory of every word you have written so far. Before choosing each new word, you can look back at all previous words — but you cannot look forward, because those words don't exist yet. That is exactly what the decoder-only transformer does: causal masking enforces the "can't see the future" constraint, while self-attention gives the model that perfect photographic memory, letting any token directly attend to any earlier token regardless of distance. The feed-forward network after attention acts like your brain's "composition instinct" — it processes the attended context through a fast lookup of pattern→continuation associations burned in during training. The residual connections are the through-line of the original input — no matter how deep the stack, there is always a highway back to the raw token signal, preventing the representational collapse that plagued deep RNNs. At inference, the KV-cache is like sticky notes: once you've read and summarized each earlier token (computed its K and V), you stick those notes to your desk and never re-read the original source — you just glance at the sticky notes and write the next word.

Historical Development

Origins · 1 minhow this was discovered

History

2014

Bahdanau, Cho, and Bengio introduce additive attention for sequence-to-sequence models, breaking the fixed-length bottleneck of vanilla encoder-decoder RNNs and planting the conceptual seed of all transformer attention.

2017

Vaswani et al. publish "Attention Is All You Need," introducing the full encoder-decoder transformer with multi-head scaled dot-product attention, positional encodings, and the Pre/Post-LayerNorm residual structure.

2018

Radford et al. (OpenAI) publish GPT-1, the first large-scale decoder-only transformer pretrained on BooksCorpus (~800M tokens) with next-token prediction, achieving strong transfer learning across NLP benchmarks via fine-tuning.

2019

Radford et al. publish GPT-2 (1.5B parameters), demonstrating that decoder-only transformers exhibit emergent zero-shot task performance — the model was initially withheld for "safety" concerns, marking the first AI release controversy.

2020

Brown et al. publish GPT-3 (175B parameters) and introduce in-context learning (ICL), showing that decoder-only transformers can perform few-shot reasoning with no gradient updates — just examples in the prompt.

2022

Hoffmann et al. (DeepMind) publish the Chinchilla scaling laws, showing most contemporary models were undertrained; Touvron et al. release LLaMA showing compute-optimal smaller decoder-only models beat larger ones if trained longer.

2023

The open-weight decoder-only ecosystem explodes: Llama 2, Mistral 7B, Falcon, and others demonstrate that decoder-only transformers at 7B–70B parameters can approach GPT-4 on many benchmarks when fine-tuned with RLHF or DPO.

2024

Mixture-of-Experts (MoE) decoder-only transformers (Mixtral 8×7B, GPT-4 reportedly MoE) demonstrate that sparsely-gated decoder-only architectures achieve frontier capability with ~1/4 of the active parameter count per forward pass.

Why It Matters

Real-World Impact

GPT-4 (OpenAI, 2023)A reportedly MoE decoder-only transformer with an estimated ~1.8T parameters; uses multi-head attention with rotary positional encodings (RoPE) and is fine-tuned with RLHF, serving as the backend for ChatGPT and the Microsoft Copilot suite used by hundreds of millions of users daily.

Llama 3 (Meta AI, 2024)An open-weight decoder-only transformer (8B–405B parameters) using grouped-query attention (GQA) to reduce KV-cache memory by sharing key/value heads across groups of query heads, enabling deployment on consumer GPUs.

GitHub Copilot (OpenAI Codex)A decoder-only transformer fine-tuned on ~159GB of public code, demonstrating that the causal language modeling objective transfers perfectly to code generation — the next-token prediction task naturally models program continuation.

AlphaCode 2 (DeepMind, 2023)A decoder-only transformer specialized for competitive programming, reaching the 85th percentile on Codeforces contests, demonstrating that causal generation can solve problems requiring multi-step algorithmic reasoning.

Gemini Ultra (Google DeepMind, 2023)A natively multimodal decoder-only transformer that ingests interleaved text, image, audio, and video tokens through a unified causal attention mechanism — showing that the decoder-only paradigm extends beyond text to arbitrary token sequences.

Claude (Anthropic, 2023–2024)Trained with Constitutional AI (CAI) and reinforcement learning from AI feedback (RLAIF) on top of a decoder-only base, demonstrating that alignment methods compose cleanly with the decoder-only architecture.

Real-Time Translation (DeepL, 2023)Production MT systems increasingly replace encoder-decoder transformers with decoder-only models prompted with source text, leveraging the simplified architecture for lower latency and easier fine-tuning on domain-specific corpora.