ALiBi Position Encoding

ALiBi (Attention with Linear Biases) is a position encoding scheme for transformer models that replaces additive positional embeddings with a fixed, non-learned linear penalty applied directly to attention logits as a function of query-key distance. It matters because it enables transformers to extrapolate to sequence lengths significantly longer than those seen during training — a property called length generalization — without any architectural modification or fine-tuning. The counterintuitive fact: ALiBi uses no learned parameters whatsoever for position information, yet outperforms learned positional embeddings on perplexity benchmarks while generalizing to sequences 2–8× longer than the training context window.

The core insight

Instead of encoding absolute or relative position into the token representations themselves, ALiBi subtracts a scalar bias from each attention score proportional to the distance between the query and key tokens — the further apart two tokens are, the more their attention score is penalized. This bias is applied after the dot product but before the softmax, so it never distorts the representational content of keys or queries. Each attention head uses a different geometric slope for its penalty, covering a spectrum of locality preferences from very local to moderately global. The "aha moment" is realizing that position information doesn't need to be embedded — it can be a recency prior baked directly into the attention mechanism as an inductive bias.

Common trapCommon Misconception

The most common mistake is assuming ALiBi is a form of relative position encoding analogous to Shaw et al. (2018) or T5's relative biases — and therefore that it learns head-specific position representations. This is wrong. ALiBi's slopes are deterministic functions of the number of heads, computed once from a closed-form geometric sequence, and are never updated by gradient descent. Practitioners then wrongly conclude that the slopes should be tuned as hyperparameters; in practice, the paper demonstrates robustness across the fixed schedule. The correct mental model is that ALiBi is a fixed inductive bias — an architectural prior about recency — not a position representation to be learned. Conflating it with learnable relative biases leads to unnecessary ablations and misattribution of failures.

How It Works

Attention Score Modification

In standard scaled dot-product attention, the pre-softmax logit between query position $i$ and key position $j$ is:

\quad e_{ij} = \frac{q_i^\top k_j}{\sqrt{d_k}}

Explanation: This is the raw compatibility score between query token $i$ and key token $j$ , scaled by the square root of the head dimension to control gradient magnitudes. Here $q_i, k_j \in \mathbb{R}^{d_k}$ are the projected query and key vectors, and $d_k$ is the head dimension. ALiBi modifies this by subtracting a linear bias:

\quad e_{ij}^{\text{ALiBi}} = \frac{q_i^\top k_j}{\sqrt{d_k}} - m \cdot |i - j|

Explanation: The attention logit is penalized by the token distance $|i-j|$ scaled by a head-specific slope $m$ , so distant tokens are systematically suppressed before the softmax without altering the query or key vectors. The quantity $|i - j|$ is the absolute distance between the query position $i$ and the key position $j$ in the sequence. Because causal language models attend only to $j \leq i$ , this simplifies to $i - j \geq 0$ , making the bias non-negative and purely subtractive. The full attention distribution for head $h$ at query position $i$ is:

\quad \alpha_{ij}^{(h)} = \text{softmax}_j\!\left(\frac{q_i^\top k_j}{\sqrt{d_k}} - m_h (i - j)\right)

Explanation: The softmax exponentiates the penalized logits and normalizes, so tokens exponentially closer to the query receive disproportionately higher weight under the recency prior.

Slope Schedule

Given $n$ attention heads, the slopes $\{m_h\}_{h=1}^{n}$ are defined as a geometric sequence:

\quad m_h = 2^{-\frac{8h}{n}}, \quad h = 1, 2, \ldots, n

Explanation: Slopes decrease geometrically, assigning exponentially different locality radii to each head — early heads impose strong recency bias while later heads remain more permissive toward distant tokens. Concretely, for $n = 8$ heads this produces slopes $\{2^{-1}, 2^{-2}, \ldots, 2^{-8}\} = \{0.5, 0.25, \ldots, 0.0039\}$ . For $n = 16$ heads, the sequence uses $2^{-0.5}, 2^{-1}, \ldots, 2^{-8}$ , i.e. steps of $8/n$ in the exponent. The key structural property is that the bias matrix for a single head with slope $m$ is:

\quad B_m = -m \begin{pmatrix} 0 & & & \\ 1 & 0 & & \\ 2 & 1 & 0 & \\ \vdots & & \ddots & 0 \end{pmatrix}

Explanation: This lower-triangular Toeplitz structure means the bias depends only on relative distance, making it equivalent to a translation-invariant recency mask that can be precomputed independently of any input. This Toeplitz structure is critical: because $B_m$ is input-independent and purely a function of position offsets, it can be precomputed once and added to every attention computation with negligible overhead.

Why Extrapolation Works

At training time the model sees sequences of length $L$ . At inference time, for a token at position $i > L$ , the bias for distant keys scales as $m \cdot (i - j)$ which can become very large, but because the model has learned to interpret the shape of the penalized attention distribution rather than absolute logit values, it degrades gracefully. Formally, the effective context window for head $h$ — the distance beyond which a key token contributes less than $\epsilon$ attention weight relative to the closest key — satisfies:

\quad d_{\text{eff}}^{(h)} \approx \frac{\log(1/\epsilon)}{m_h}

Explanation: Heads with small slopes $m_h$ have exponentially larger effective windows, so the ensemble of heads collectively covers a wide range of distances even when extrapolating beyond training length. This means the multi-head ensemble naturally distributes "attention budget" across distance scales, and the inductive bias remains well-defined at any sequence length without positional out-of-distribution issues that plague sinusoidal or learned embeddings.

researcher · Ofir Press | b. 1991 | Inventor of ALiBi and lead author of the 2022 ICLR paper

Ofir Press completed his PhD at the University of Washington and later joined the University of Edinburgh and Meta AI Research, working on efficient and generalizable language model architectures. His specific contribution to ALiBi was the insight that a fixed linear penalty — rather than any learned or sinusoidal embedding — was sufficient to confer strong length generalization, formalized in "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (Press et al., ICLR 2022). A surprising fact: Press first tested the idea informally during a single weekend experiment, and the perplexity numbers were compelling enough that the core mechanism was never substantially changed between that prototype and the published version.

Figure

ALiBi Attention Bias Matrix and Multi-Head Slope Schedule

Intuition

The Mental Model

Imagine you are reading a long document and trying to understand the current word. Most of the time, words nearby are far more relevant than words from dozens of sentences ago — this is the recency prior that ALiBi encodes. Think of each attention head as a spotlight operator who dims the lights on the stage based on how far an actor stands from center: the further away, the dimmer. Crucially, the dimming function is not learned — it is physically built into the spotlight hardware — so the operator can still illuminate actors at any distance; they are just systematically less prominent. Different heads use different dimmer profiles (from aggressive to gentle), so together the ensemble can represent both tightly local dependencies and moderately long-range ones. When you extend the performance to a bigger stage than rehearsed, the dimmer still works perfectly because it only cares about relative distance, not about whether you