Positional Encodings

A Transformer layer computes attention scores from dot products between query and key vectors derived from token embeddings. If you permute the input tokens and permute the output labels correspondingly, the attention mechanism's output is permuted the same way — it does not "notice" that anything changed. Formally, self-attention is a permutation-equivariant function: $\text{Attn}(P X) = P\,\text{Attn}(X)$ for any permutation matrix $P$. This equivariance is a direct consequence of the same weight matrices $W_Q, W_K, W_V$ being applied identically at every position, with no notion of position built into the computation itself. A positional encoding is a vector $\mathrm{PE}(p) \in \mathbb{R}^d$ associated with each sequence index $p = 0, 1, 2, \dots$, that is combined with the token embedding $E(x_p)$ — almost always by addition — to form the actual input to the Transformer:

Because $\mathrm{PE}(p)$ differs across positions, the combined vector $h_p$ breaks the symmetry that made pure token embeddings permutation-invariant. Attention now operates on vectors that encode both content and location, and the model can, in principle, learn to attend differently to "the third token" versus "the tenth token," or to prefer nearby tokens over distant ones. There are two broad design families. Sinusoidal (fixed, non-learned) encodings, introduced in the original Transformer paper, define $\mathrm{PE}(p)$ using sine and cosine functions of geometrically increasing wavelengths — no parameters are trained for this piece at all. Learned (absolute) positional embeddings instead treat $\mathrm{PE}(p)$ as a row of a trainable embedding matrix, exactly analogous to a word embedding table but indexed by position instead of vocabulary id. A third, increasingly dominant family — relative positional schemes such as RoPE (Rotary Position Embedding) and ALiBi — abandon the "absolute position vector added at the input" pattern entirely and instead inject position information directly into the attention score computation, which foreshadows design choices used throughout the Transformer Encoder Stack.

Sinusoidal encoding. For a model dimension $d$ (even), position $p$, and dimension index $i \in \{0, \dots, d/2 - 1\}$, define:

Each pair of dimensions $(2i, 2i+1)$ oscillates at its own frequency $\omega_i = 10000^{-2i/d}$. Low-index pairs ($i$ small) oscillate rapidly (short wavelength), giving fine-grained information about nearby positions; high-index pairs oscillate slowly (long wavelength), giving coarse information that distinguishes far-apart positions. The base $10000$ is a hyperparameter chosen so that wavelengths range from about $2\pi$ up to about $10000 \cdot 2\pi$, comfortably covering typical sequence lengths.

Learned encoding. Maintain a trainable matrix $W_{\text{pos}} \in \mathbb{R}^{L_{\max} \times d}$, where $L_{\max}$ is the maximum sequence length the model supports. Position $p$'s encoding is simply row $p$ of this matrix, updated by gradient descent along with every other parameter. Both are added to the token embedding at the bottom of the network, once, before the first attention layer: ``` function build_input_representations(tokens, embedding_table, pos_encoding_fn, d_model):

# tokens: list of token ids, length L

# embedding_table: vocab_size x d_model matrix

# pos_encoding_fn: function(position, d_model) -> vector in R^d_model

# (either the sinusoidal formula or a lookup into a

# learned position-embedding table) L = length(tokens) H = zero_matrix(L, d_model) for p in 0 .. L-1: token_vec = embedding_table[tokens[p]] # content pos_vec = pos_encoding_fn(p, d_model) # position H[p] = token_vec + pos_vec # combine return H

# H is fed as the initial layer input to the Transformer stack ``` ``` function sinusoidal_pe(p, d_model, base=10000): vec = zero_vector(d_model) for i in 0 .. (d_model/2 - 1): freq = 1.0 / (base ** (2i / d_model)) vec[2i] = sin(p freq) vec[2i + 1] = cos(p * freq) return vec ``` Once $H$ is built, ordinary self-attention layers proceed unchanged: queries, keys, and values are linear projections of $H$, and attention weights come from scaled dot products. The positional signal now rides inside every query/key dot product, so attention scores implicitly depend on the relative offset between positions — this is the entire mechanism; nothing about the attention formula itself needs to change.

Time and space complexity. Computing sinusoidal encodings for a sequence of length $L$ with model dimension $d$ takes $O(L \cdot d)$ time and $O(L \cdot d)$ space if materialized, though in practice the values can be precomputed once (they don't depend on the input tokens or on training) and cached or even generated on the fly with $O(1)$ extra memory per position beyond the $d$-dimensional output. Learned encodings cost the same $O(L \cdot d)$ to look up, plus $O(L_{\max} \cdot d)$ parameters that must be stored and trained — an asymptotically small addition compared to the $O(d^2)$ parameters in each attention/feed-forward layer, but a real cost that scales with the maximum supported sequence length. Both schemes add negligible overhead compared to the $O(L^2 d)$ cost of self-attention itself, so positional encoding is essentially free relative to the rest of the Transformer's computation.

Why sinusoidal encodings let attention recover relative position. The key mathematical property is that $\mathrm{PE}(p+k)$ can be written as a linear function of $\mathrm{PE}(p)$ for any fixed offset $k$, with coefficients depending only on $k$ (not on $p$). This follows from the angle-addition identities:

For each frequency pair $i$, this is exactly a $2\times 2$ rotation matrix applied to $(\sin(\omega_i p), \cos(\omega_i p))$:

Since this rotation matrix depends only on $k$, a linear projection (such as the query/key projections $W_Q, W_K$ inside attention) can in principle learn to extract relative-offset information from the difference between two positions' encodings, without ever seeing $p$ or $p+k$ explicitly labeled. This is the mathematical justification behind the oft-repeated claim that "sinusoidal encodings let the model generalize to relative positions" — it is not a vague intuition but a concrete linear-algebraic fact about the encoding's structure.

Correctness in the sense of injectivity. For a positional encoding scheme to do its job, it must at minimum assign (near-)distinct vectors to distinct positions within the range the model will see, so that attention can actually distinguish them. Sinusoidal encodings satisfy this as long as $L \le$ the range over which the chosen frequencies avoid aliasing — with $d/2$ frequencies spanning wavelengths from $2\pi$ to $10000 \cdot 2\pi$, positions up to several thousand remain distinguishable in practice, though at very large $L$ the lowest frequencies barely change and encodings for very distant positions can become numerically close along those slow dimensions. Learned encodings are trivially injective by construction (distinct rows), but only exist for $p < L_{\max}$; there is no principled way to compute $\mathrm{PE}(p)$ for $p \ge L_{\max}$, which is precisely the extrapolation failure mode discussed below.

Tradeoffs, honestly stated. Sinusoidal encodings need no training and can (in theory) be evaluated at positions beyond those seen during training, giving some extrapolation ability, but empirically Transformers trained on fixed-length sequences still struggle to generalize well to much longer ones, since the attention weights that use these encodings were only ever trained on the shorter-range patterns. Learned encodings are more flexible and often perform as well or better within the trained range, but cannot handle positions past $L_{\max}$ at all, and they add real (if modest) parameter count. Neither purely-additive scheme is perfectly suited to relative reasoning, which is why relative-position and rotary schemes (RoPE, ALiBi, T5's relative bias) have become the default in many current large-scale models — they inject position directly into the attention score $q_i \cdot k_j$ as a function of $i - j$, sidestepping some of the extrapolation weaknesses of absolute encodings while keeping the $O(L^2 d)$ attention cost unchanged.

Let's trace sinusoidal encodings by hand for a tiny model with $d_{\text{model}} = 4$ (so two frequency pairs, $i = 0, 1$) and base $10000$. Frequencies: $\omega_0 = 10000^{-0/4} = 1$, and $\omega_1 = 10000^{-2/4} = 10000^{-0.5} = 1/100 = 0.01$.

Observe two things directly in the table. First, dimensions 0–1 (fast frequency, $\omega_0=1$) change dramatically from position to position — this pair is sensitive to local order, distinguishing token 1 from token 2 sharply. Second, dimensions 2–3 (slow frequency, $\omega_1 = 0.01$) barely move across these four positions — this pair only becomes informative for distinguishing positions that are hundreds apart. Together, the fast and slow pairs give the model a multi-resolution "clock" for position, much like how a digital clock's seconds hand and hour hand jointly encode both fine and coarse time. Now check the rotation identity for offset $k=1$ using dimensions 0–1 at $p=1 \to p=2$:

Wait — let's compute numerically with $\cos(1)=0.540$, $\sin(1)=0.841$:

This matches $\mathrm{PE}(2,0)=0.909$ and $\mathrm{PE}(2,1)=-0.416$ up to rounding — confirming that the encoding at $p=2$ is exactly the rotation-by-$k{=}1$ image of the encoding at $p=1$, as the angle-addition math predicted. For contrast, a learned encoding scheme in this toy setup would simply store four independent trainable vectors $W_{\text{pos}}[0], \dots, W_{\text{pos}}[3]$, initialized randomly (e.g., small Gaussian noise) and updated by backpropagation through the language-modeling loss — no rotation structure is guaranteed, and the model must discover any useful relative-position structure purely from data and gradient descent.

Positional encoding as a formal component of attention-based sequence models was popularized by the "Attention Is All You Need" paper, which proposed sinusoidal encodings specifically so that the model could be trained without recurrence or convolution while still respecting word order.

Every major Transformer-based system depends on some positional encoding scheme to make word order matter. BERT and its descendants use learned absolute position embeddings up to a fixed maximum sequence length. GPT-style autoregressive language models historically used learned positional embeddings as well, while many recent large language models — including LLaMA-family and similar open models — use RoPE, which rotates query and key vectors as a function of position directly inside the attention dot product, giving strong relative-position behavior and better length extrapolation in practice. Encoder-decoder machine translation systems, the original use case for sinusoidal encodings, still rely on the same core insight even as the specific formula has been superseded in many production systems. Understanding positional encodings is a prerequisite for understanding how the full Transformer Encoder Stack processes ordered text, and it connects directly to broader questions in Sequence Modeling about how architectures represent structure without explicit recurrence.

Ask the tutor: How does Rotary Position Embedding mathematically guarantee that attention scores depend only on relative position, and why does that improve length generalization compared to sinusoidal encodings?