Full Transformer Architecture

A sequence-to-sequence (seq2seq) task maps an input sequence $x_1, \dots, x_n$ to an output sequence $y_1, \dots, y_m$, where $n$ and $m$ can differ and are not known in advance. The full Transformer solves this with two cooperating stacks. The Transformer Encoder Stack reads the entire source sequence at once and produces a set of contextualized vectors $h_1, \dots, h_n$, one per input token, each aware of the whole input through self-attention. The Transformer Decoder Stack generates the output sequence one token at a time, conditioning at each step on all previously generated tokens and on the encoder's output. The component that makes this a genuine encoder-decoder model, rather than two independent stacks, is cross-attention (also called encoder-decoder attention). In each decoder layer, after masked self-attention over the target prefix, a second attention sub-layer computes queries from the decoder's current hidden states but keys and values from the encoder's final output. This lets the decoder ask, at every generation step, "which parts of the source sentence are most relevant to the word I am about to produce?" Because attention is computed over all source positions simultaneously, this reference is direct — unlike a recurrent encoder-decoder model, there is no information bottleneck forcing the entire source meaning through a single fixed-size vector. Two attention masking conventions distinguish the stacks. Encoder self-attention is unmasked (bidirectional): position $i$ can attend to every position $j$, since the whole input is available at once. Decoder self-attention is causally masked: position $i$ can attend only to positions $j \le i$, since at generation time future tokens do not exist yet. Cross-attention has no masking restriction on the source side (all source positions are visible) but is only ever queried from decoder positions, never the reverse. This asymmetry — full visibility into the past and the source, no visibility into the future — is what makes the model both a valid probabilistic language model over the target and a genuine translator of the source.

End-to-end pipeline. Training operates on a paired example $(x, y)$. The encoder processes $x$ once, producing $H \in \mathbb{R}^{n \times d}$. The decoder is fed the target sequence shifted right (prepended with a start token, ``), and — using teacher forcing — is trained to predict the true next token at every position simultaneously, in one parallel forward pass, using $H$ as the cross-attention source at every layer. At inference time there is no ground-truth target to feed, so tokens are generated one at a time: each newly produced token is appended to the decoder input and the whole process (or an incrementally cached version of it) repeats until an end token is produced or a length limit is reached. ``` function ENCODE(x[1..n]): E = Embed(x) + PositionalEncoding(1..n) # n x d H = E for l in 1..L_enc: A = SelfAttention(H, H, H, mask = None) # bidirectional H = LayerNorm(H + A) F = FeedForward(H) H = LayerNorm(H + F) return H # n x d, final encoder memory function DECODE_TRAIN(y[1..m], H):

# teacher forcing: full target fed at once, causal mask enforces left-to-right validity D = Embed(shift_right(y)) + PositionalEncoding(1..m) # m x d for l in 1..L_dec: A1 = SelfAttention(D, D, D, mask = CAUSAL) # target attends only to its own past D = LayerNorm(D + A1) A2 = CrossAttention(query=D, key=H, value=H, mask = None) # target attends to all of source D = LayerNorm(D + A2) F = FeedForward(D) D = LayerNorm(D + F) logits = D @ W_out.T # m x |V| return logits function DECODE_INFER(H, max_len, bos, eos): tokens = [bos] for t in 1..max_len: D = DECODE_ONE_STEP(tokens, H) # same layer stack, causal mask, cross-attn to H next_token = argmax_or_sample(D[-1] @ W_out.T) tokens.append(next_token) if next_token == eos: break return tokens[1:] # drop bos ``` The three attention operations that appear above (encoder self-attention, decoder self-attention, cross-attention) all use the same scaled dot-product formula, differing only in where queries, keys, and values come from and whether a mask is applied: ``` function ATTENTION(Q, K, V, mask): scores = (Q @ K.T) / sqrt(d_k) # (len_Q x len_K) if mask is not None: scores = scores + mask # mask entries are 0 or -infinity weights = softmax(scores, axis=-1) # rows sum to 1 return weights @ V # (len_Q x d_v) ``` For decoder self-attention, `mask[i][j] = 0` if $j \le i$ else $-\infty$, so that after softmax, position $i$ places zero probability on any future position $j > i$. For cross-attention, `Q` comes from the decoder's current layer, `K` and `V` come from the encoder's final output `H`, and there is no mask — every source position is eligible. The figure below shows the full information flow for one decoding step, including where the two masking regimes and the cross-attention bridge sit relative to each other.

Time complexity. Let $n$ be source length, $m$ be target length, $d$ the model dimension, and $L$ the number of layers per stack (assume roughly equal for encoder and decoder). Each self-attention sub-layer costs $O(n^2 d)$ (encoder) or $O(m^2 d)$ (decoder) for computing all pairwise scores and their weighted sums, since the score matrix has $O(n^2)$ or $O(m^2)$ entries and each requires an $O(d)$ dot product; each feed-forward sub-layer costs $O(n d^2)$ or $O(m d^2)$. Cross-attention costs $O(nm d)$: the decoder's $m$ queries each attend over $n$ encoder positions, and each score/weighted-sum computation is $O(d)$. Summing over $L$ layers, total training-time compute for one example is

For typical settings where $d$ is comparable to or larger than sequence length, the $nd^2$ and $md^2$ feed-forward terms often dominate in practice, but the $n^2$, $m^2$, and $nm$ attention terms are what make long sequences expensive — this is the well-known quadratic bottleneck of standard attention.

Training vs. inference asymmetry. During training, teacher forcing means the entire target sequence is available up front, so all $m$ decoder positions are processed in parallel in one forward pass — the causal mask is just an additive matrix, not a sequential control-flow constraint. This is why Transformer training parallelizes far better than recurrent seq2seq models, whose recurrence forces $O(m)$ sequential steps. At inference, no such shortcut exists: each output token depends on the previous one through the argmax/sampling step, so generation genuinely requires $m$ sequential decoder passes, each — without caching — costing $O(m^2 d + nmd)$ redundant recomputation of earlier positions. Production systems avoid most of this redundancy with a key-value cache: since keys and values for already-generated positions never change, they are computed once and reused, reducing each incremental step to $O(md + nd)$ work rather than recomputing the full attention matrix, so total inference cost becomes $O(m(md+nd))=O(m^2d + nmd)$ overall — the same asymptotic total as an uncached forward pass over the growing sequence, but spread correctly across steps rather than repeated wastefully at each one.

Space complexity. Storing all activations, attention weight matrices, and the KV cache costs $O(n^2 + m^2 + nm)$ for attention matrices (per layer, per head) and $O((n+m)d)$ for hidden states and the cache, times $L$ layers and $H$ heads for the attention matrices specifically: $O(LH(n^2+m^2+nm))$.

Correctness argument. Two properties must hold for the model to define a valid, learnable conditional distribution $P(y \mid x)$. Autoregressive validity (no label leakage). The chain rule factorization $P(y\mid x) = \prod_{t=1}^m P(y_t \mid y_{<t}, x)$ requires that the computation of $P(y_t \mid \cdot)$ never depends on $y_t, y_{t+1}, \dots$. The causal mask enforces this as an invariant: by induction on layer depth, if the input to layer $l$ at position $t$ depends only on target positions $\le t$ (true trivially for the embedded input, since position $t$'s embedding is $y_{t-1}$ under the right-shift), then after masked self-attention — which by construction assigns zero weight to any position $> t$ — the output at position $t$ still depends only on positions $\le t$. Cross-attention and feed-forward layers act position-wise (cross-attention reads from the encoder, which has no dependence on $y$ at all), so they preserve the invariant. Hence at every layer and every position $t$, the representation is a function only of $y_{<t}$ and $x$, and the final softmax at position $t$ computes exactly $P(y_t \mid y_{<t}, x)$ — matching training-time teacher forcing to inference-time autoregressive generation exactly. Full-input visibility for the encoder. Because encoder self-attention is unmasked, $h_i$, the encoder's representation of position $i$, is influenced by every $x_j$, not just $x_{\le i}$. This is correct and necessary for encoding: unlike generation, understanding a completed input sentence has no causality constraint, so restricting the encoder would only discard useful context (this is precisely why encoder-only models like BERT can be trained with bidirectional objectives, while decoder-only generative models cannot). Tradeoff. The price of this correctness is the quadratic attention cost above, plus the strict sequential nature of inference — even with KV caching, generation cannot be parallelized across time steps, unlike encoding, because each token's distribution genuinely depends on the sampled identity of all previous tokens.

Consider translating the English sentence "the cat sat" into French, target "le chat était assis" (simplified here to "le chat s'assit" for brevity — we'll trace generating "le chat" as the first two tokens). Source tokens: $x = [\text{the}, \text{cat}, \text{sat}]$, $n=3$.

Step 1 — Encoding (done once). The encoder processes all three tokens simultaneously with bidirectional self-attention, producing $H = [h_\text{the}, h_\text{cat}, h_\text{sat}]$. Note $h_\text{cat}$ is influenced by both "the" (its determiner) and "sat" (its verb) — full sentence context, computed in a single pass.

Step 2 — Decoding step 1. Decoder input so far: `[]`. Masked self-attention over a length-1 sequence trivially attends only to `` itself. Cross-attention: the query derived from ``'s representation attends over $\{h_\text{the}, h_\text{cat}, h_\text{sat}\}$; because French typically opens a noun phrase with a determiner, attention weight concentrates on $h_\text{the}$. The output distribution places high probability on "le". Suppose argmax selects "le".

Step 3 — Decoding step 2. Decoder input: `[, le]`. Causal self-attention at the current (second) position attends over `{, le}` only — position 2 cannot see any hypothetical future token. Cross-attention now shifts weight toward $h_\text{cat}$, since the model needs the noun following the determiner. Output distribution favors "chat".

Step 4 — Decoding step 3. Decoder input: `[, le, chat]`. Cross-attention shifts weight toward $h_\text{sat}$ to produce the verb; self-attention at position 3 sees `{, le, chat}` and correctly excludes any not-yet-generated token. This continues until `` is emitted.

Contrast this with training on this same pair: instead of four sequential calls, the decoder receives `[, le, chat, s'assit]` all at once, and the causal mask ensures position 2's output logits (used to compute the loss against "chat") never had access to "chat" or "s'assit" as inputs, so the parallel computation exactly reproduces what four sequential inference-style steps would have computed — this equivalence is precisely the invariant argued above.

:::warning[The single most dangerous mistake is omitting or misplacing the causal mask in decoder self-attention. If unmasked, training loss will look deceptively good — the model achieves near-perfect next-token prediction because it can literally see the answer — while inference-time generation, which cannot cheat, collapses into incoherent or repetitive output. This bug is silent: no shape error, no crash, just a model that "trained fine" and generates nonsense, often discovered late in a project.]:::

The full encoder-decoder Transformer was introduced by a team at Google Brain and Google Research, building on a decade of work in neural sequence-to-sequence modeling and attention mechanisms that preceded it, including Bahdanau-style additive attention for RNN encoder-decoders and the insight that attention alone, without recurrence, could carry long-range dependencies more directly.

:::physicist[Ashish Vaswani|b. 1980s (contemporary)|Lead author of "Attention Is All You Need" (2017), which proposed replacing recurrence entirely with self-attention and cross-attention, establishing the full encoder-decoder Transformer as the architecture underlying nearly all subsequent large-scale sequence models.]:::

The full encoder-decoder Transformer underlies production machine translation systems, including large-scale services that translate between dozens of languages in real time. It is the backbone of abstractive summarization systems, where an encoder digests a long document and a decoder produces a condensed summary while cross-attending to the source. Speech recognition and speech synthesis pipelines use the same pattern, with the encoder consuming audio features and the decoder producing text (or vice versa). Multimodal systems that generate image captions or answer questions about images reuse the identical cross-attention bridge, substituting a vision encoder for a text encoder while keeping the decoder unchanged. The text-to-text framework popularized by models like T5 casts nearly every NLP task — classification, translation, summarization, question answering — as a single encoder-decoder problem, directly exploiting this architecture's generality. Understanding this full pipeline is also the necessary foundation for the Pretraining-Finetuning Paradigm, since decisions about which half of the architecture to keep, and what objective to pretrain it with, define the major families of modern pretrained language models.

Ask the tutor: How would the architecture change if you needed the decoder to attend to two different encoders at once, such as one for text and one for an image?