T5: Text-to-Text Transfer Transformer

Unified Text-to-Text Framework

T5 maps every task to the format:

Explanation: The model finds the output token sequence $\hat{y}$ that maximizes the conditional probability given the concatenation of the task prefix string and the input $x$, with $\theta$ denoting all model parameters. Here $\oplus$ denotes string concatenation, `prefix` is a human-readable task descriptor like `"summarize:"`, and $y$ is generated autoregressively token by token. No architectural change is made between tasks — only the prefix changes.

Encoder-Decoder Architecture

T5 uses a standard Transformer encoder-decoder. The encoder computes contextual representations:

Explanation: At each encoder layer $l$, the output is the layer-normalized sum of the multi-head self-attention output and the residual from the previous layer $H^{(l-1)}$, building progressively richer contextual representations. T5 makes one notable architectural modification: it replaces absolute sinusoidal positional encodings with relative position biases, added directly to attention logits:

Explanation: The attention score between query position $i$ and key position $j$ is the standard scaled dot product augmented by $b(i-j)$, a learned scalar bias that depends only on the relative offset between positions, enabling better generalization to sequence lengths not seen during training. Here $q_i \in \mathbb{R}^{d_k}$ and $k_j \in \mathbb{R}^{d_k}$ are query and key vectors, $d_k$ is the head dimension, and $b : \mathbb{Z} \to \mathbb{R}$ is a learned function bucketed into 32 bins that collapse distant positions.

Span-Corruption Pretraining Objective

Unlike BERT's token-level masking, T5 masks contiguous spans of tokens. Given input tokens $x_{1:n}$, spans are replaced with sentinel tokens $\langle S_i \rangle$, and the decoder must reconstruct all masked spans:

Explanation: The pretraining loss is the negative log-likelihood summed over all masked spans $i$ and all tokens $t$ within each span, conditioned on the sentinel token, previously predicted tokens within the span, and the full encoder hidden states $H_{\text{enc}}$. The span corruption rate is 15% of tokens, with a mean span length of 3 tokens, empirically chosen via ablation over the C4 dataset.

Cross-Attention in the Decoder

The decoder attends to encoder outputs via cross-attention at every decoder layer:

Explanation: The decoder's query matrix $Q$ (from the decoder's masked self-attention) attends over the encoder's key and value matrices $K, V$ (both derived from $H_{\text{enc}}$), with projection matrices $W_Q, W_K, W_V \in \mathbb{R}^{d_\text{model} \times d_k}$, producing a weighted combination of encoder information at each decoding step.

PyTorch Usage

```python from transformers import T5ForConditionalGeneration, T5Tokenizer model = T5ForConditionalGeneration.from_pretrained("t5-base") tokenizer = T5Tokenizer.from_pretrained("t5-base")

# Summarization — just change the prefix for any task input_text = "summarize: The quick brown fox jumps over the lazy dog. " \ "The dog barely reacted to the fox's agile leap." inputs = tokenizer(input_text, return_tensors="pt", max_length=512, truncation=True) outputs = model.generate( inputs.input_ids, max_length=50, num_beams=4, early_stopping=True ) print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# → "The quick brown fox jumps over the lazy dog."

# Fine-tuning with teacher forcing labels = tokenizer("fox jumps over dog", return_tensors="pt").input_ids loss = model(**inputs, labels=labels).loss loss.backward() ``` The `labels` tensor is automatically shifted right inside `T5ForConditionalGeneration`; sentinel tokens for pretraining are handled by `T5Tokenizer`'s `create_sentinel_ids` API.