Sliding Window Attention

Sliding Window Attention (SWA) is a sparse attention mechanism in which each token attends only to a fixed-size local neighborhood of preceding and following tokens, reducing the quadratic $O(n^2)$ complexity of full self-attention to $O(n \cdot w)$ where $w$ is the window size. It matters because it makes Transformer-based language models tractable on sequences of tens of thousands to millions of tokens without sacrificing the representational power needed for long-document reasoning. The counterintuitive fact: despite each token seeing only $w \ll n$ neighbors locally, information propagates globally across the full sequence depth in $O(n/w)$ layers — meaning a model with sufficient depth recovers full receptive field coverage with a fraction of the compute.

The core insight

Full attention is a dense all-pairs comparison that scales catastrophically with sequence length; sliding window attention replaces this dense graph with a sparse banded graph where each token communicates only with its $w$ nearest neighbors per layer. The "aha moment" is recognizing that locality is sufficient per layer because stacking $L$ layers gives an effective receptive field of $L \cdot w/2$ tokens on each side — exactly as convolutions build global context from local filters. This means the architectural choice is not "local vs. global" but rather "where should global mixing happen": deep in the stack through propagation, or immediately through explicit global tokens or cross-window connections.

Common trapCommon Misconception

The most widespread mistake is assuming that sliding window attention with window size $w$ limits the model's receptive field to $w$ tokens permanently. Practitioners read "each token attends to $w$ neighbors" and conclude the model cannot capture dependencies longer than $w$ tokens — and then cargo-cult excessively large window sizes, negating the efficiency gains. The correct mental model is that receptive field grows linearly with depth: after $L$ layers, token $i$ has transitively received information from tokens in $[i - L\cdot w/2,\; i + L\cdot w/2]$ . In Longformer (Beltagy et al., 2020), a 12-layer model with $w=512$ achieves a receptive field of 3,072 tokens on each side — covering most practical document lengths. The fix: reason about effective receptive field, not per-layer window size.

How It Works

Formal Setup

Let $\mathbf{X} \in \mathbb{R}^{n \times d}$ be the input sequence of $n$ tokens with embedding dimension $d$ . In standard scaled dot-product attention, the attention matrix $\mathbf{A} \in \mathbb{R}^{n \times n}$ is fully dense. Sliding window attention imposes a sparsity mask $\mathbf{M} \in \{0, -\infty\}^{n \times n}$ such that only entries within a band of half-width $w/2$ are retained. The masked attention score matrix is:

\quad \mathbf{S}_{ij} = \frac{\mathbf{q}_i^\top \mathbf{k}_j}{\sqrt{d_k}} + \mathbf{M}_{ij}, \quad \mathbf{M}_{ij} = \begin{cases} 0 & |i - j| \leq w/2 \\ -\infty & \text{otherwise} \end{cases}

Explanation: each query $\mathbf{q}_i$ only receives finite scores from keys $\mathbf{k}_j$ within the window, and $-\infty$ pre-softmax effectively zeros out all out-of-window attention weights. Here $\mathbf{q}_i = \mathbf{x}_i W^Q$ , $\mathbf{k}_j = \mathbf{x}_j W^K$ are the query and key projections with $W^Q, W^K \in \mathbb{R}^{d \times d_k}$ , and $d_k$ is the key dimension. The output for token $i$ is then:

\quad \mathbf{o}_i = \sum_{j:\,|i-j|\leq w/2} \frac{\exp(\mathbf{S}_{ij})}{\sum_{j':\,|i-j'|\leq w/2} \exp(\mathbf{S}_{ij'})} \cdot \mathbf{v}_j

Explanation: the output at position $i$ is a locally-normalized weighted sum of value vectors $\mathbf{v}_j = \mathbf{x}_j W^V$ restricted to the window, so attention weights are re-normalized within each window rather than globally.

Complexity Analysis

The total number of attended pairs is at most $n \cdot w$ (compared to $n^2$ for dense attention), giving:

\quad \text{FLOPs}_{\text{SWA}} = O(n \cdot w \cdot d_k) \quad \text{vs.} \quad \text{FLOPs}_{\text{full}} = O(n^2 \cdot d_k)

Explanation: for fixed window size $w \ll n$ , the compute scales linearly with sequence length, making million-token contexts computationally tractable on commodity hardware.

Effective Receptive Field

After $L$ attention layers each with window $w$ , the maximum token distance over which information can flow is bounded by:

\quad \text{ERF}(L, w) = L \cdot \frac{w}{2}

Explanation: this is a worst-case lower bound on receptive field — in practice, residual connections and layer normalization can allow faster information mixing, but $L \cdot w/2$ gives the strict graph-theoretic propagation depth assuming causal paths of length $L$ .

Dilated Sliding Window Attention

To expand the receptive field without increasing $w$ , one can apply a dilation factor $r$ so that the attended positions form a strided pattern:

\quad \mathbf{M}_{ij}^{(\text{dilated})} = \begin{cases} 0 & |i - j| \leq r \cdot w/2 \;\text{ and }\; (i-j) \equiv 0 \pmod{r} \\ -\infty & \text{otherwise} \end{cases}

Explanation: with dilation $r$ , each window covers a span of $r \cdot w$ tokens while still attending to only $w$ positions, so the receptive field grows by a factor of $r$ at no additional compute cost — analogous to dilated/atrous convolutions in CNNs.

researcher · Iz Beltagy | b. 1986 | Longformer and Sliding Window Attention for Long Documents

Iz Beltagy is a research scientist at the Allen Institute for AI (AI2), where his work focuses on making large language models efficient and applicable to scientific and long-form text. He led the development of Longformer (2020), which popularized sliding window attention combined with task-specific global tokens as a practical recipe for processing documents up to 4,096+ tokens with near-linear complexity. Surprisingly, Longformer's design drew direct inspiration from WaveNet's dilated convolutions — Beltagy's team explicitly framed the attention pattern as a "dilated sliding window" to build intuition for ML practitioners already familiar with convolutional architectures.

researcher · Ashish Vaswani | b. 1983 | Original Transformer and Attention Mechanisms

Ashish Vaswani was a senior researcher at Google Brain and is widely recognized as the primary architect of the original Transformer paper "Attention Is All You Need" (2017), which introduced scaled dot-product attention and multi-head attention as the foundation for modern sequence models. Although the original Transformer used full $O(n^2)$ attention, Vaswani's clean formalization of query-key-value attention created the modular scaffolding that made sparse variants like sliding window attention straightforward to derive and implement. He later co-founded Essential AI after a period at Google, and notably the 2017 paper was written during an intensive sprint that Vaswani has described as one of the most collaborative experiences of his research career.

Figure

Sliding Window Attention: Sparsity Pattern and Information Propagation

Intuition

The Mental Model

Imagine you are a journalist trying to understand a very long document — say, a 500-page legal brief. You don't read the entire document before forming any opinion about page 200; instead, you read locally — the paragraph you're on, the previous few paragraphs, and the next few — and carry a running mental summary forward. After enough pages of this local reading, your understanding of page 200 is informed by everything that came before, but the mechanism was always local. Sliding window attention is exactly this: each token reads a small neighborhood at each layer, but through the residual stream, meaning accumulates and propagates so that deeper layers carry distant context already compressed into the representations of nearby tokens. The global picture emerges from stacked local views — the same principle behind how pixels in a deep CNN "see" the whole image even though each convolutional kernel is tiny.

Historical Development

Origins · 1 minhow this was discovered

History

1989

Yann LeCun's convolutional networks establish the key precedent that local receptive fields stacked across layers can encode global structure — the conceptual ancestor of sliding window attention.

1997

Hochreiter & Schmidhuber's LSTM introduces gated local-in-time state propagation, demonstrating that bounded-horizon mechanisms can carry unbounded-range information through learned gating.

2017

Vaswani et al., "Attention Is All You Need" — introduces full

O(n^2)

scaled dot-product attention as the universal sequence mixer; establishes the attention formalism that sparse variants would later constrain.

2018

Liu et al. introduce "Generating Wikipedia by Summarizing Long Sequences," identifying the quadratic attention bottleneck as the primary barrier to processing book-length documents with Transformers.

2019

Child et al., "Generating Long Sequences with Sparse Transformers" (OpenAI) — formally introduce block-local and strided sparse attention patterns, directly prefiguring sliding window attention and achieving

O(n\sqrt{n})

complexity; models trained on sequences of 12,288 tokens.

2020

Beltagy, Peters & Cohan, "Longformer: The Long-Document Transformer" (AI2) — systematizes sliding window attention with dilated windows and per-task global tokens; demonstrates BERT-level NLU performance on documents up to 4,096 tokens with linear complexity. Longformer becomes the practical standard.

2020

Zaheer et al., "Big Bird: Transformers for Longer Sequences" (Google Research) — proves theoretically that sparse attention (including sliding window plus global plus random) is a universal approximator of full attention and is Turing-complete, providing the first rigorous expressiveness guarantees for SWA-style mechanisms.

2023

Jiang et al., "Mistral 7B" — demonstrates that sliding window attention with

w=4096

within a grouped-query framework achieves state-of-the-art performance among open 7B models on long-context benchmarks, bringing SWA into mainstream production LLM training.

2024

Gemini 1.5 (Google DeepMind) and other frontier models incorporate hybrid architectures alternating sliding window and full attention layers across multi-million token contexts, establishing this hybrid as the dominant design pattern for long-context frontier models.

Why It Matters

Real-World Impact

Longformer (Beltagy et al., 2020, AI2)Processes full scientific papers and legal documents up to 4,096 tokens with linear complexity using

w=512

sliding window plus task-specific global tokens; achieves state-of-the-art on WikiHop QA, TriviaQA, and Hyperpartisan classification — tasks that require cross-paragraph reasoning impossible with BERT's 512-token limit.

Mistral 7B (Mistral AI, 2023)Uses sliding window attention with

w=4096

and a rolling KV cache that retains only the

w

most recent key-value pairs, enabling efficient inference on sequences far longer than the training window through the observation that SWA naturally implements a form of streaming decoding with bounded memory.

Big Bird (Google Research, 2020)Combines SWA with random global tokens to process genomic sequences (up to 4,096 nucleotides) for DNA binding site prediction and gene expression tasks — domains where long-range base-pair interactions are biologically critical and the

O(n^2)

barrier was previously prohibitive.

Sparse Transformer (OpenAI, 2019)Applied strided sliding window attention to generate coherent 64×64 images (pixel-by-pixel) and raw audio waveforms at 12,288-step sequences, demonstrating that creative generation quality is preserved under aggressive attention sparsification.

Gemini 1.5 Pro (Google DeepMind, 2024)Employs a hybrid architecture where earlier Transformer layers use sliding window attention for local feature extraction and later layers use full attention with efficient approximations, enabling demonstrated 1M+ token context windows with strong needle-in-haystack retrieval performance.

Clinical NLP SystemsSWA-based models are deployed in hospital systems to process full patient records (often thousands of tokens spanning years of