Build Anthropic's Contextual Retrieval pattern

Introduction

When you split a long document into chunks and embed them in isolation, each chunk loses awareness of the surrounding document it came from — a snippet that reads "set timeout to 30 seconds" looks identical to every other timeout in the corpus, and retrieval surfaces the wrong source. Teams that ship RAG pipelines without preserving chunk-to-document context see retrieval accuracy plateau no matter how strong the embedding model is, and downstream answers cite the wrong evidence. In this lesson you will build Anthropic's Contextual Retrieval pattern end-to-end — generating per-chunk context summaries with an LLM, prepending them before embedding, and extending BM25 over the same context-augmented text — so that semantic and keyword search both benefit from document-level grounding. By the end you'll have a working ContextGenerator, ContextualChunker, and ContextualBM25 that together reproduce the pipeline Anthropic reported as a 67% reduction in retrieval failures when paired with reranking.

Key Terminology

• Contextual Retrieval: Anthropic's pattern where each chunk is prefixed with an LLM-generated summary of how it fits in its parent document before being embedded and indexed.
• Contextual Embedding: A dense vector produced from the concatenation of context + original_chunk, so semantic search matches against both the chunk's content and its document-level scope.
• Contextual BM25: A sparse keyword index built over the same context-augmented chunk text, so lexical matches can fire on terminology that appears in the context summary even when absent from the raw chunk.

Concepts

Contextual Retrieval is a composition of three ideas. First, chunk-level grounding: for every chunk you send the full parent document plus the chunk to an LLM and ask for a 50–100 word sentence describing where the chunk fits — this sentence becomes the chunk's "context prefix". Second, dual-index reuse of the same augmented text: the prefixed string is what gets embedded and what gets fed into BM25, so semantic search and keyword search share a single, document-aware view of each chunk. Third, separation of stored text from indexed text: the contextual prefix exists to improve recall, but the original chunk text is preserved alongside it so reranking and final display still operate on the author's words, not on LLM-generated framing.

Code Walkthrough

Generating and Prepending Chunk-Specific Context

Send each chunk along with its parent document to an LLM, then prepend the returned context sentence to the chunk before it is embedded:

Code snippetpython
1from openai import OpenAI
2
3class ContextGenerator:
4    CONTEXT_PROMPT = """You are a document analysis assistant. Given the full document
5and a specific chunk from that document, generate a concise context sentence (50-100 words)
6explaining where this chunk fits within the document's structure and topic.
7
8<document>
9{document}
10</document>
11
12<chunk>
13{chunk}
14</chunk>
15
16Respond with ONLY the context sentence, nothing else."""
17
18    def __init__(self, model: str = "gpt-4o-mini"):
19        self.client = OpenAI()
20        self.model = model
21
22    def generate_context(self, document: str, chunk: str) -> str:
23        response = self.client.chat.completions.create(
24            model=self.model,
25            messages=[
26                {
27                    "role": "user",
28                    "content": self.CONTEXT_PROMPT.format(
29                        document=document[:8000],
30                        chunk=chunk,
31                    ),
32                }
33            ],
34            max_tokens=150,
35            temperature=0.0,
36        )
37        return response.choices[0].message.content.strip()
38
39class ContextualChunker:
40    def __init__(
41        self,
42        base_chunker: "RecursiveChunker",
43        context_generator: ContextGenerator,
44    ):
45        self.chunker = base_chunker
46        self.generator = context_generator
47
48    def chunk_with_context(self, document: str) -> list[dict]:
49        base_chunks = self.chunker.chunk(document)
50        contextual_chunks = []
51        for chunk in base_chunks:
52            context = self.generator.generate_context(document, chunk["text"])
53            contextual_chunks.append({
54                "text": f"{context}\n\n{chunk['text']}",
55                "original_text": chunk["text"],
56                "context": context,
57                "index": chunk["index"],
58                "token_count": chunk["token_count"],
59            })
60        return contextual_chunks

• ContextGenerator: The prompt template sends the full document (truncated to 8000 characters to stay within context limits) alongside the specific chunk and demands ONLY a 50-100 word context sentence. temperature=0.0 maximizes consistency across chunks from the same document; max_tokens=150 prevents overly verbose summaries that would dilute the chunk's embedding.
• ContextualChunker: Wraps any base chunking strategy and produces records where text is context + "\n\n" + original_chunk (the string you embed and BM25-index) while original_text preserves the author's words for reranking and display. For production pipelines, swap the sequential loop for async calls or the OpenAI Batch API for ~50% cost reduction.

Implementing Contextual BM25

Extend keyword indexing to include terms from context summaries:

1from collections import defaultdict
2import math
3
4class ContextualBM25:
5    def __init__(self, k1: float = 1.2, b: float = 0.75):
6        self.k1 = k1
7        self.b = b
8        self.doc_count = 0
9        self.avg_doc_len = 0.0
10        self.doc_freqs = defaultdict(int)
11        self.index = {}
12
13    def index_chunks(self, chunks: list[dict]):
14        for chunk in chunks:
15            text = chunk.get("text", "")
16            terms = text.lower().split()
17            doc_id = chunk.get("index", 0)
18
19            self.index[doc_id] = {
20                "terms": terms,
21                "length": len(terms),
22                "chunk": chunk,
23            }
24
25            unique_terms = set(terms)
26            for term in unique_terms:
27                self.doc_freqs[term] += 1
28
29        self.doc_count = len(self.index)
30        total_len = sum(d["length"] for d in self.index.values())
31        self.avg_doc_len = total_len / max(self.doc_count, 1)
32
33    def search(self, query: str, top_k: int = 10) -> list[dict]:
34        query_terms = query.lower().split()
35        scores = {}
36
37        for doc_id, doc_data in self.index.items():
38            score = 0.0
39            term_freqs = defaultdict(int)
40            for t in doc_data["terms"]:
41                term_freqs[t] += 1
42
43            for term in query_terms:
44                if term not in term_freqs:
45                    continue
46                tf = term_freqs[term]
47                df = self.doc_freqs.get(term, 0)
48                idf = math.log((self.doc_count - df + 0.5) / (df + 0.5) + 1)
49                numerator = tf * (self.k1 + 1)
50                denominator = tf + self.k1 * (
51                    1 - self.b + self.b * doc_data["length"] / self.avg_doc_len
52                )
53                score += idf * numerator / denominator
54
55            if score > 0:
56                scores[doc_id] = score
57
58        ranked = sorted(scores.items(), key=lambda x: -x[1])[:top_k]
59        return [
60            {"doc_id": doc_id, "score": score, "chunk": self.index[doc_id]["chunk"]}
61            for doc_id, score in ranked
62        ]

The combination of Contextual Embeddings (for semantic search) and Contextual BM25 (for keyword search) with a reranker produces the full Contextual Retrieval pipeline that achieved Anthropic's published 67% improvement in retrieval accuracy. You'll know it works when the same query returns the chunk whose context summary — not just its raw text — matches the query's document-level intent.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Embed and BM25-index the context + original_chunk string, but keep the original chunk text on the record so reranking and display use the author's words.
2. ✓Pin the context-generation LLM call to temperature=0.0 and cap it with max_tokens=150 so context prefixes stay consistent and don't dilute the chunk's embedding.
3. ✓Truncate the parent document passed into the context prompt (e.g. first 8000 characters) to stay within model context limits while still grounding the chunk in document-level scope.

Don'ts

1. ✗Don't embed raw chunks alongside contextual chunks in the same index — mixing augmented and un-augmented vectors breaks the recall gains the pattern is designed to produce.
2. ✗Don't let the LLM return commentary or formatting around the context sentence; the prompt must demand the sentence only, or the prefix will pollute embeddings.
3. ✗Don't skip Contextual BM25 and rely on Contextual Embeddings alone — Anthropic's published 67% improvement requires the hybrid combination plus a reranker, not embeddings in isolation.