Sequence-to-Sequence Models

Encoder

The encoder reads the source sequence $\mathbf{x} = (x_1, x_2, \ldots, x_T)$ token by token. At each step $t$, an embedding lookup converts the token index into a dense vector, which is fed into a recurrent cell (GRU or LSTM) to update the hidden state:

Explanation: The encoder hidden state at time $t$ is a deterministic function of the previous hidden state and the current token's embedding — it accumulates a running summary of the source sequence.

Symbols:

• $\mathbf{h}_t^{(\text{enc})} \in \mathbb{R}^d$ — encoder hidden state at step $t$, dimensionality $d$
• $f_\theta$ — the recurrent cell (GRU/LSTM) parameterized by weights $\theta$
• $\mathbf{E}_{\text{src}} \in \mathbb{R}^{d \times |V_{\text{src}}|}$ — source embedding matrix, $|V_{\text{src}}|$ is source vocabulary size
• $x_t \in \mathbb{Z}$ — integer token index at position $t$

After processing all $T$ tokens, the final hidden state $\mathbf{c} = \mathbf{h}_T^{(\text{enc})}$ becomes the context vector — the compressed encoding of the entire source.

Decoder

The decoder generates the target sequence $\mathbf{y} = (y_1, y_2, \ldots, y_{T'})$ autoregressively. Its hidden state evolves as:

Explanation: The decoder state at step $t$ depends on the previous decoder state, the embedding of the previously generated token, and the fixed context vector — the context is re-injected at every step.

Symbols:

• $\mathbf{s}_t \in \mathbb{R}^{d'}$ — decoder hidden state at step $t$
• $g_\phi$ — decoder recurrent cell with separate parameters $\phi$
• $\mathbf{E}_{\text{tgt}}$ — target-side embedding matrix
• $y_{t-1}$ — the token generated at the previous step (or `` at $t=1$)
• $\mathbf{c}$ — context vector from encoder

Output Distribution

At each decoding step the hidden state is projected onto the target vocabulary via a linear layer followed by a softmax:

Explanation: This converts the decoder's internal state into a probability distribution over all target vocabulary words, from which the next token is sampled or argmax-selected.

Symbols:

• $\mathbf{W}_o \in \mathbb{R}^{|V_{\text{tgt}}| \times d'}$ — output projection weight matrix
• $\mathbf{b}_o \in \mathbb{R}^{|V_{\text{tgt}}|}$ — output bias
• $y_{<t} = (y_1, \ldots, y_{t-1})$ — all previously generated tokens

Training Objective

The model is trained by minimizing the cross-entropy loss summed over all target positions, averaged across a batch of $N$ examples:

Explanation: The model is penalized for assigning low probability to the correct next token at every position in every target sequence — equivalently, it is maximizing the likelihood of the ground-truth translations.

Symbols:

• $N$ — batch size
• $T'_n$ — length of the $n$-th target sequence
• $y_t^{(n)}$ — the ground-truth token at position $t$ for example $n$

During training, teacher forcing is used: the ground-truth token $y_{t-1}^{(n)}$ (not the model's own prediction) is fed as input at each decoder step, which stabilizes gradients but introduces a train/inference discrepancy known as exposure bias.

PyTorch Sketch

```python import torch import torch.nn as nn class Encoder(nn.Module): def __init__(self, vocab_size, emb_dim, hid_dim): super().__init__() self.embed = nn.Embedding(vocab_size, emb_dim, padding_idx=0) self.rnn = nn.GRU(emb_dim, hid_dim, batch_first=True) def forward(self, src): # src: (B, T) embedded = self.embed(src) # (B, T, emb_dim) outputs, hidden = self.rnn(embedded) # outputs: (B, T, hid_dim) return outputs, hidden # hidden: (1, B, hid_dim) class Decoder(nn.Module): def __init__(self, vocab_size, emb_dim, hid_dim): super().__init__() self.embed = nn.Embedding(vocab_size, emb_dim, padding_idx=0) self.rnn = nn.GRU(emb_dim + hid_dim, hid_dim, batch_first=True) self.fc_out = nn.Linear(hid_dim, vocab_size) def forward(self, tgt_tok, hidden, context): # tgt_tok: (B,1), context: (B,1,hid_dim) emb = self.embed(tgt_tok) # (B, 1, emb_dim) rnn_in = torch.cat([emb, context], dim=2)# (B, 1, emb_dim+hid_dim) out, hidden = self.rnn(rnn_in, hidden) # out: (B, 1, hid_dim) logits = self.fc_out(out.squeeze(1)) # (B, vocab_size) return logits, hidden ```