Chunking Strategies for RAG

Formal Problem Setup

Let a corpus $\mathcal{D}$ consist of $N$ documents. A chunking function $\chi$ maps each document $d_i$ to an ordered sequence of text spans:

Explanation: Each document is decomposed into an ordered list of non-overlapping (or partially overlapping) text chunks, where the total number of chunks $K_i$ depends on both the document length and the chosen strategy. Each chunk is then encoded by an embedding model $\phi : \mathcal{T} \to \mathbb{R}^d$ mapping text to a $d$-dimensional dense vector:

Explanation: The embedding function projects the discrete token sequence of a chunk into a continuous vector space where geometric proximity approximates semantic similarity. At query time, a user query $q$ is embedded and the top-$r$ chunks are retrieved by approximate nearest neighbor search using cosine similarity:

Explanation: Cosine similarity measures the angle between two unit-normalized vectors; chunks whose embedding direction aligns most closely with the query embedding are considered most relevant. The core trade-off can be framed as a bias–variance decomposition of retrieval quality. Let $\ell$ denote average chunk length in tokens. Retrieval precision $P(\ell)$ and recall $R(\ell)$ trade off with chunk size:

Explanation: This loss captures the fundamental tension — larger chunks reduce recall loss (the answer is more likely present) but increase precision loss (irrelevant context pollutes the embedding); $\alpha$ is a task-specific weight. For overlapping chunks with stride $s$ and window $w$ (both in tokens), the overlap ratio $\rho$ determines context continuity:

Explanation: When $\rho = 0$ there is no overlap (pure partitioning); as $\rho \to 1$ the stride approaches zero and adjacent chunks are nearly identical, maximizing context continuity at the cost of index size. ---

Strategy Taxonomy

#### 1. Fixed-Size Chunking The simplest strategy: split every document into windows of exactly $w$ tokens with stride $s$. ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2") def fixed_size_chunks(text: str, window: int = 256, stride: int = 32) -> list[str]: """ Tokenize text and produce overlapping fixed-size chunks. window: chunk size in tokens stride: number of tokens to advance between chunks """ token_ids = tokenizer.encode(text, add_special_tokens=False) chunks = [] for start in range(0, len(token_ids), stride): end = start + window chunk_ids = token_ids[start:end] if len(chunk_ids) < window // 4: # discard tail slivers break chunks.append(tokenizer.decode(chunk_ids)) if end >= len(token_ids): break return chunks ```

Pros: $O(N)$ complexity, deterministic, easy to reason about index size. Cons: Splits sentences and paragraphs mid-thought; embedding quality degrades when semantic boundaries are violated. --- #### 2. Recursive Character / Structure-Aware Chunking Splits on a priority hierarchy of delimiters: `["\n\n", "\n", ". ", " ", ""]`. At each level, if a piece exceeds `max_size`, it is recursively split on the next delimiter. ```python from langchain.text_splitter import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter( chunk_size=512, # characters, not tokens chunk_overlap=64, separators=["\n\n", "\n", ". ", " ", ""], length_function=len, ) chunks = splitter.split_text(document_text) ``` This is the most widely used baseline in production RAG systems because it respects natural language boundaries without requiring document-format parsing. --- #### 3. Semantic Chunking Groups sentences together as long as their consecutive embedding similarity exceeds a threshold $\theta$, creating variable-length chunks whose boundaries coincide with genuine topic shifts. ```python import numpy as np from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") def semantic_chunks(sentences: list[str], threshold: float = 0.85) -> list[str]: embeddings = model.encode(sentences, normalize_embeddings=True)

# cosine similarity between consecutive sentences sims = (embeddings[:-1] * embeddings[1:]).sum(axis=1) chunks, current = [], [sentences[0]] for i, sim in enumerate(sims): if sim >= threshold: current.append(sentences[i + 1]) else: chunks.append(" ".join(current)) current = [sentences[i + 1]] if current: chunks.append(" ".join(current)) return chunks ```

Pros: Chunk boundaries correspond to semantic shifts — much better embedding quality. Cons: Requires a full encoding pass over all sentences before chunking; $O(N)$ embedding calls. --- #### 4. Document-Structure Chunking (Markdown / HTML-aware) Uses document hierarchy (headers, sections, list items) as natural chunk boundaries. Particularly effective for technical documentation, wikis, and legal documents with explicit section structure. ```python from langchain.text_splitter import MarkdownHeaderTextSplitter headers_to_split_on = [ ("#", "H1"), ("##", "H2"), ("###", "H3"), ] md_splitter = MarkdownHeaderTextSplitter( headers_to_split_on=headers_to_split_on, strip_headers=False, ) md_chunks = md_splitter.split_text(markdown_document)

# Each chunk carries metadata: {"H1": "Introduction", "H2": "Background"} ``` Metadata retention is a key advantage: each chunk knows its structural provenance, enabling metadata filtering at retrieval time. --- #### 5. Agentic / Proposition-Level Chunking Decomposes documents into atomic propositions — indivisible factual statements — using an LLM. Introduced by Chen et al. (2023) in the Dense X Retrieval paper, this produces the finest-grained retrieval units. ```python from openai import OpenAI client = OpenAI() def proposition_chunk(passage: str) -> list[str]: prompt = ( "Decompose the following passage into a list of atomic, " "self-contained factual propositions. Each proposition must be " "independently understandable. Return one proposition per line.\n\n" f"Passage: {passage}" ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": prompt}], temperature=0.0, ) return [line.strip() for line in response.choices[0].message.content.strip().split("\n") if line.strip()] ```

Pros: Maximum retrieval precision; no irrelevant co-text dilutes the embedding. Cons: LLM inference cost at index time; proposition count can explode for long corpora. --- #### 6. Hierarchical / Parent-Child Chunking Maintains two index levels: small child chunks for high-precision retrieval, large parent chunks (or full documents) for context-rich generation. After retrieval, child chunks are "expanded" to their parent before being passed to the LLM. ```python from langchain.retrievers import ParentDocumentRetriever from langchain.storage import InMemoryStore from langchain.vectorstores import Chroma from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain.embeddings import HuggingFaceEmbeddings child_splitter = RecursiveCharacterTextSplitter(chunk_size=200) parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000) vectorstore = Chroma( embedding_function=HuggingFaceEmbeddings( model_name="sentence-transformers/all-MiniLM-L6-v2" ) ) docstore = InMemoryStore() retriever = ParentDocumentRetriever( vectorstore=vectorstore, docstore=docstore, child_splitter=child_splitter, parent_splitter=parent_splitter, ) ``` This strategy decouples retrieval granularity from generation context, directly addressing the precision–recall trade-off in Equation [4]. ---

---

---

---

Evaluation Framework

A rigorous chunking evaluation measures three complementary metrics:

The RAGAS framework (Es et al., 2023) provides automated LLM-based scoring for all three. ```python from ragas import evaluate from ragas.metrics import context_precision, context_recall, faithfulness from datasets import Dataset eval_data = Dataset.from_dict({ "question": questions, "answer": generated_answers, "contexts": retrieved_chunks, # list of list of str "ground_truth": reference_answers, }) results = evaluate( dataset=eval_data, metrics=[context_precision, context_recall, faithfulness], ) print(results.to_pandas()) ```