Reranking in RAG Pipelines

Reranking in Retrieval-Augmented Generation (RAG) pipelines is a second-stage scoring step that reorders an initial set of candidate documents retrieved by a fast first-stage retriever, using a more computationally expensive but semantically accurate model to maximize the relevance of context passed to the language model. It matters because first-stage retrievers (sparse BM25 or dense bi-encoders) optimise for recall at scale, not precision, and the quality of the final generated answer is tightly coupled to the relevance of the top-k documents actually fed into the prompt. The counterintuitive fact: a reranker that sees only the top-20 candidates from a cheap retriever routinely outperforms an expensive cross-encoder applied exhaustively to the entire corpus, because the ranking problem is much easier than the retrieval problem.

The core insight

First-stage retrieval and second-stage reranking solve fundamentally different problems under fundamentally different computational budgets. A bi-encoder embeds a query and every document independently, enabling sub-linear search via ANN indices, but sacrifices the token-level cross-attention that captures subtle semantic relationships. A reranker receives the query and a candidate document together as a single input, letting every query token attend to every document token, which is why it is far more accurate — but also why you can only afford to run it on O(100) candidates, not millions.

How It Works

Reranking reformulates document scoring as a pointwise or pairwise classification problem. Given a query qq and a candidate document did_i, a cross-encoder reranker computes a scalar relevance score by concatenating the two sequences and passing them through a transformer:

1
si=fθ([CLS]  q  [SEP]  di  [SEP])\quad s_i = f_\theta([CLS]\; q\; [SEP]\; d_i\; [SEP])

Explanation: The model reads query and document jointly in a single forward pass, letting every token attend to every other token, and extracts a scalar relevance score from the [CLS] embedding. Here fθf_\theta is typically a linear head on top of the [CLS] hidden state of a pretrained language model (e.g. BERT, DeBERTa), and si∈Rs_i \in \mathbb{R} is the raw logit. Documents are then sorted in descending order of sis_i and the top-kk are selected for the prompt. Training uses a pairwise ranking loss. Given a positive document d+d^+ and a negative d−d^- for query qq:

2
Lpair=max⁡ ⁣(0,  1−s++s−)\quad \mathcal{L}_{\text{pair}} = \max\!\left(0,\; 1 - s^+ + s^-\right)

Explanation: This margin-based hinge loss penalises any case where the negative document scores within 1 unit of the positive, pushing the model to rank relevant documents strictly above irrelevant ones. A softer, more widely used alternative is the listwise softmax cross-entropy over the NN candidates returned by the first stage:

3
Llist=−log⁡exp⁡(s+)∑j=1Nexp⁡(sj)\quad \mathcal{L}_{\text{list}} = -\log \frac{\exp(s^+)}{\sum_{j=1}^{N} \exp(s_j)}

Explanation: This InfoNCE-style loss treats the single positive document as the correct class among all N candidates, encouraging the reranker to assign it the highest probability in the softmax distribution. At inference, the full reranking pipeline computes a normalised probability for each candidate:

4
P(di∣q)=exp⁡(si/τ)∑j=1Nexp⁡(sj/τ)\quad P(d_i \mid q) = \frac{\exp(s_i / \tau)}{\sum_{j=1}^{N} \exp(s_j / \tau)}

Explanation: Dividing by temperature τ\tau controls score sharpness — a small τ\tau concentrates mass on the top document, while a larger τ\tau produces a softer distribution suitable for score-weighted fusion. For Reciprocal Rank Fusion (RRF), a training-free alternative that merges rankings from multiple retrievers before reranking:

5
RRF(d)=∑r∈R1k+rankr(d)\quad \text{RRF}(d) = \sum_{r \in \mathcal{R}} \frac{1}{k + \text{rank}_r(d)}

Explanation: Each retriever rr contributes a score inversely proportional to the document's rank in its ordered list, with constant kk (typically 60) dampening the influence of very high ranks and preventing single-retriever domination.

researcher · Jimmy Lin | b. 1976 | Cascade Retrieval Architecture

Jimmy Lin is a Professor at the University of Waterloo whose research spans information retrieval and NLP, and he was instrumental in formalising the cascaded multi-stage ranking paradigm that underlies modern RAG pipelines — his work on the MS MARCO leaderboard directly shaped how the community benchmarks first-stage retrieval and second-stage reranking as separate, composable components. His 2021 paper "A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques" helped practitioners reason about the retrieval-reranking tradeoff in terms of interaction depth vs. computational cost. Surprisingly, Lin started his career studying computational biology before pivoting entirely to information retrieval after a chance encounter with Lucene in graduate school.

Figure
RAG Pipeline with Reranking — Two-Stage Funnel
RAG Pipeline: Two-Stage Retrieval + Reranking Funnel Query "What is RAG?" Bi-Encoder ANN Index top-100 recall Candidates d₁, d₂, … d₁₀₀ High recall Low precision Cross-Encoder Reranker score(q, dᵢ) → sᵢ joint attention Top-k Docs d₁, d₃, d₇ k = 3..5 LLM Generator Grounded answer generation Stage 1: Retrieve O(millions docs) Stage 2: Rerank O(100 candidates) 100 → 5 Vector Store millions of docs embed
Figure
Cross-Encoder vs Bi-Encoder Attention Pattern
Attention Patterns: Bi-Encoder vs Cross-Encoder Bi-Encoder Query Tower q₁ q₂ q₃ eq Document Tower d₁ d₂ d₃ ed eq · ed ✗ No cross-token attention Fast — ANN indexable Cross-Encoder [CLS] query [SEP] document [SEP] CLS q₁ q₂ SEP d₁ d₂ d₃ SEP Full self-attention over all tokens × L layers score s ✓ Full cross-token attention Slow — O(100) candidates only

Intuition

The Mental Model

Imagine you're a hiring manager for a senior engineering role. You receive 10,000 applications, so you first ask HR to screen by keyword matching — anyone without "Python" or "distributed systems" in their résumé gets filtered out. This is your first-stage retriever: fast, coarse, high recall. You end up with 100 finalists. Now you personally read each of those 100 résumés in the context of the specific job description — noticing which candidates actually built distributed systems versus just mentioning the word, which projects are truly relevant versus tangentially related. You're doing cross-attention: the job description and the résumé interact token-by-token in your head. This is the reranker. The brilliant insight is that you never needed to personally read all 10,000 applications — that would have been impossible and unnecessary. HR already handled the impossible part. You just needed to be precise about the 100 survivors. A third analogy: reranking is like a wine sommelier's second pour. The first pour (retrieval) eliminates wines from the wrong region or vintage year with a quick look at the label. The second pour (reranking) is a slow, deliberate tasting — contextual, nuanced, expensive per sample, but applied only to the shortlist.

Historical Development

Origins · 1 minhow this was discovered
History
1994
Robertson & Sparck Jones formalise BM25, establishing the probabilistic relevance model that dominates first-stage retrieval for the next three decades — still the default first stage in many RAG systems.
2003
Joachims introduces pairwise learning-to-rank via SVMs in "Optimizing Search Engines using Clickthrough Data," laying the mathematical foundation for training ranking models from implicit feedback.
2010
Burges et al. at Microsoft Research release LambdaRank and LambdaMART, making gradient-boosted tree rerankers the state of the art on commercial web search and establishing the cascade retrieval paradigm.
2019
Nogueira & Cho publish "Passage Re-ranking with BERT," demonstrating that a fine-tuned BERT cross-encoder dramatically outperforms BM25 + traditional learning-to-rank on MS MARCO, catalysing the modern neural reranking era.
2020
Hofstätter et al. introduce TK-Sparse and Efficient Neural Ranking; simultaneously, ColBERT (Khattab & Zaharia) proposes late interaction as a middle ground between bi-encoder speed and cross-encoder accuracy.
2021
MS MARCO leaderboard proliferates specialised rerankers (monoT5, DuoBERT); Cohere releases a hosted reranking API, making cross-encoder reranking accessible without GPU infrastructure.
2022
Sentence Transformers library standardises the train-a-cross-encoder workflow in PyTorch; LLM-based rerankers emerge, using instruction-tuned models to output relevance judgements as text.
2023
RankGPT (Sun et al.) demonstrates that GPT-4 can rerank documents via listwise prompting with zero training, achieving competitive performance on BEIR; LLaMA-based open-source rerankers follow.
2024
BGE-Reranker-v2 and Jina Reranker v2 release multilingual cross-encoder rerankers; reranking becomes a first-class component in LangChain, LlamaIndex, and Haystack RAG frameworks with dedicated reranker abstractions.

Why It Matters

Real-World Impact
Perplexity AIUses a two-stage pipeline where a fast retriever fetches web snippets and a cross-encoder reranker selects the 3-5 most relevant passages to inject into the prompt, directly improving citation accuracy and factual grounding in answers.
Cohere Rerank APIProvides a hosted cross-encoder (rerank-english-v3.0) that enterprises integrate as a drop-in second stage; internal benchmarks show 20-35% improvement in nDCG@10 over BM25