Positional Encoding

Sinusoidal Positional Encoding (Vaswani et al., 2017)

The original formulation adds a fixed, deterministic vector to each token embedding. For a sequence position $pos$ and embedding dimension index $i$, the encoding is defined across the full $d_{\text{model}}$-dimensional space by alternating sine and cosine functions at geometrically decreasing frequencies:

Explanation: The even-indexed dimensions of the positional vector for token at position `pos` are set to a sine wave whose frequency decreases exponentially as the dimension index `i` increases.

Explanation: The odd-indexed dimensions mirror the even ones with a cosine instead of sine, giving each frequency band a two-dimensional sinusoidal basis — analogous to encoding an angle as both its sine and cosine to make it uniquely recoverable.

Symbol definitions:

• $pos \in \{0, 1, \ldots, L-1\}$: integer token position in the sequence
• $i \in \{0, 1, \ldots, d_{\text{model}}/2 - 1\}$: dimension pair index
• $d_{\text{model}}$: total embedding dimensionality (e.g., 512 in the base Transformer)
• $10000$: the wavelength base; chosen so wavelengths range from $2\pi$ (highest frequency) to $10000 \cdot 2\pi$ (lowest)

The final input to the first Transformer layer is the sum of the token embedding and its positional encoding:

Explanation: Token embedding and positional signal live in the same $d_{\text{model}}$-dimensional space, so they can be added directly — the model learns to disentangle the two sources of information through its projection matrices.

The Relative Offset Property

A key reason sinusoidal encodings are theoretically appealing is that $PE(pos + k)$ can be expressed as a linear function of $PE(pos)$ for any fixed offset $k$. Specifically, using angle-addition identities:

Explanation: The positional encoding at any offset `k` from position `pos` is a rotation of the encoding at `pos` — meaning attention weight dot-products can in principle learn to compute relative distances without ever explicitly seeing position integers. This linearity means the dot product $PE(pos)^\top PE(pos+k)$ depends only on $k$, not on the absolute value of $pos$, giving the model a built-in geometric structure that mirrors how we think about sequence relationships.

Learned Absolute Positional Embeddings

An alternative: treat each position index as a token, maintain a learnable embedding table $\mathbf{P} \in \mathbb{R}^{L_{\max} \times d_{\text{model}}}$, and add $\mathbf{P}[pos]$ to the token embedding. BERT and GPT-2 both use this approach. The downside is a hard sequence-length cap of $L_{\max}$ and no guaranteed generalization beyond it. ```python import torch import torch.nn as nn import math class SinusoidalPositionalEncoding(nn.Module): def __init__(self, d_model: int, max_len: int = 5000, dropout: float = 0.1): super().__init__() self.dropout = nn.Dropout(p=dropout)

# Build the PE matrix once; shape [max_len, d_model] pe = torch.zeros(max_len, d_model) position = torch.arange(0, max_len).unsqueeze(1).float() # [max_len, 1] div_term = torch.exp( torch.arange(0, d_model, 2).float() (-math.log(10000.0) / d_model) ) # [d_model/2] pe[:, 0::2] = torch.sin(position div_term) # even dims pe[:, 1::2] = torch.cos(position * div_term) # odd dims pe = pe.unsqueeze(0) # [1, max_len, d_model] self.register_buffer("pe", pe) # not a parameter def forward(self, x: torch.Tensor) -> torch.Tensor:

# x: [batch, seq_len, d_model] x = x + self.pe[:, :x.size(1), :] return self.dropout(x)

class LearnedPositionalEncoding(nn.Module): def __init__(self, d_model: int, max_len: int = 512): super().__init__() self.embedding = nn.Embedding(max_len, d_model) def forward(self, x: torch.Tensor) -> torch.Tensor: positions = torch.arange(x.size(1), device=x.device).unsqueeze(0) return x + self.embedding(positions) ```

Rotary Positional Embedding (RoPE)

Introduced by Su et al. (2021) and used in LLaMA, GPT-NeoX, and many modern LLMs, RoPE encodes position by rotating query and key vectors in 2D subspaces rather than adding a fixed vector to token embeddings. For a query vector $\mathbf{q}$ at position $m$ and key vector $\mathbf{k}$ at position $n$, the attention score becomes:

Explanation: By applying a rotation matrix $R_m$ to queries and $R_n$ to keys before computing their dot product, the resulting score depends only on the relative offset $(m-n)$, natively giving the model relative position sensitivity without modifying the architecture. Here $R_m$ is a block-diagonal rotation matrix constructed from the same sinusoidal frequencies, applied directly to the head dimension of Q and K projections. RoPE has become the dominant positional encoding scheme in large-scale language models as of 2024.