Build hybrid retrieval (semantic + BM25 + reranking)

Introduction

When you ship a RAG system that relies on semantic embeddings alone, you watch queries like "ERR_CONN_REFUSED troubleshooting" return vaguely related networking prose instead of the exact runbook the user needs — embeddings smear precise tokens into nearby concepts. Flip to BM25 keyword search alone and the opposite failure appears: paraphrased queries like "how to fix connection issues" miss documents that say "resolve network failures." Teams that ship one retrieval mode and call it done discover, on the first production traffic spike, that recall collapses for whichever query class they didn't optimize — and the LLM downstream confidently hallucinates over the wrong context.

By the end of this lesson you'll be able to build a production HybridRetriever class that runs pgvector cosine similarity and PostgreSQL BM25 full-text search concurrently through asyncio.gather(), fuses both ranked lists with reciprocal rank fusion (RRF), and returns a deduplicated, score-annotated chunk list ready for downstream reranking.

Key Terminology

• Reciprocal Rank Fusion (RRF): a rank-based merge that scores each chunk as the sum of 1 / (k + rank) across every retrieval list it appears in, sidestepping the need to calibrate cosine similarity and BM25 scores onto a common scale.
• pgvector cosine search: the PostgreSQL vector extension's <=> operator, which ranks rows by cosine distance between a query embedding and a stored vector(1536) column, accelerated by an HNSW index.
• PostgreSQL BM25 (ts_rank): the lexical search path built on tsvector + plainto_tsquery, ranking matches by ts_rank over a GIN-indexed full-text column for exact term overlap with the query.

Concepts

Why hybrid outperforms either retrieval mode alone

The core insight behind hybrid retrieval is that semantic embeddings and lexical term matching make independent errors. When a user queries "LlamaIndex workflow timeout configuration," the embedding model captures the concept of configuring timeouts in orchestration frameworks, retrieving chunks about async step timeouts, workflow retry logic, and event loop configuration. The BM25 index, operating on exact token overlap, retrieves chunks that literally contain "LlamaIndex," "workflow," and "timeout"—including configuration snippets that the embedding model might rank lower because their surrounding prose is sparse. The two result sets partially overlap on the best chunks and diverge on the marginal ones. Reciprocal rank fusion exploits this divergence: chunks that appear high in both lists receive amplified scores, while chunks that appear in only one list receive moderate scores proportional to their rank position.

The mathematical foundation of RRF is deliberately simple. For a chunk appearing at rank r in a result list, its RRF contribution from that list is 1 / (k + r), where k is a smoothing constant (typically 60). The final score for each chunk sums its RRF contributions across all result lists. This formulation is rank-based rather than score-based, which means it sidesteps the calibration problem—cosine similarity scores from pgvector and BM25 relevance scores from PostgreSQL live on incompatible scales, but their rank orderings are directly comparable.

Tuning hybrid retrieval for production workloads

Three parameters dominate hybrid retrieval quality. First, the per-source top_k values control how many candidates each retrieval path contributes. Setting these too low (under 10) starves the fusion stage of diversity; setting them too high (over 50) adds latency without meaningful recall improvement. Start with 20 per source and adjust based on RAGAS context recall measurements from another goal. Second, the RRF constant k controls how rapidly scores decay with rank. The standard value of 60 works well for most corpora, but domains with highly skewed relevance distributions (where the top-1 result is almost always correct) benefit from lower values like 20 that amplify top-rank positions. Third, the embedding model dimensionality affects both storage cost and similarity discrimination—text-embedding-3-small at 1536 dimensions offers a strong cost-quality tradeoff, but you can reduce to 512 dimensions via the model's native dimensions parameter if storage pressure dominates.

When the HybridRetriever returns its fused results, the next pipeline stage—implemented as a LlamaIndex Workflow step in another goal—applies a cross-encoder reranker (such as cross-encoder/ms-marco-MiniLM-L-6-v2) to re-score the top-N chunks using full query-chunk attention. This two-stage architecture (cheap retrieval → expensive reranking) is the standard production pattern: the retriever casts a wide net with low latency, and the reranker applies expensive but accurate scoring to a small candidate set. The agentic RAG agent in another goal can then inspect the reranked results and, if quality is insufficient, reformulate the query and trigger another retrieval cycle through the same HybridRetriever—making the retriever a reusable building block across both single-shot and iterative RAG pipelines.

Code Walkthrough

Now that you have the conceptual model for why hybrid retrieval works and how RRF fuses ranked lists, the next step is to express those ideas as runnable code. The walkthrough below moves from the data-flow diagram, to the HybridRetriever class that orchestrates both searches, to the PostgreSQL schema that backs them.

Architecture of the hybrid retrieval pipeline

Before examining code, study how a single user query fans out into parallel retrieval paths, merges at the fusion layer, and produces a unified ranked list. The following diagram traces data flow from the incoming query through embedding generation, dual-path search, and rank fusion.

This Mermaid flowchart maps the dual-retrieval pipeline that makes hybrid RAG outperform either search strategy alone. A user query forks into two paths: text-embedding-3-small generates a dense vector for pgvector cosine similarity search (embedding <=> $1), while plainto_tsquery drives PostgreSQL full-text search ranked by ts_rank. Both result sets merge through Reciprocal Rank Fusion (RRF) with k=60, where score = Σ 1/(k+r) balances semantic understanding against exact keyword matches. After deduplication by chunk_id, the pipeline emits a single ranked chunk list ready for downstream reranking.

Loading diagram...

Both search paths execute concurrently inside asyncio.gather(), so the total latency equals the slower of the two queries rather than their sum. In practice, pgvector approximate nearest neighbor search with an HNSW index completes in 5–15 ms for tables under 1M rows, and PostgreSQL GIN-indexed full-text search completes in 2–8 ms, yielding a combined wall-clock time under 20 ms for the retrieval stage.

Implementing the HybridRetriever class

The HybridRetriever class encapsulates both retrieval strategies behind a single retrieve() coroutine. The constructor accepts an asyncpg.Pool for non-blocking database access and an AsyncOpenAI client for embedding generation. The class exposes three key methods: semantic_search() generates an embedding via OpenAI's text-embedding-3-small model and queries the pgvector embedding column using the cosine distance operator <=>; bm25_search() converts the query string into a PostgreSQL tsquery and ranks matching rows by ts_rank; and retrieve() orchestrates both searches in parallel and applies reciprocal rank fusion to their results. Each method returns a list of RetrievedChunk dataclass instances carrying the chunk text, metadata, source-specific score, and rank position.

Code snippetpython
1import asyncio
2from dataclasses import dataclass, field
3from openai import AsyncOpenAI
4import asyncpg
5
6@dataclass
7class RetrievedChunk:
8    chunk_id: str
9    text: str
10    metadata: dict = field(default_factory=dict)
11    score: float = 0.0
12    rank: int = 0
13    source: str = ""
14
15class HybridRetriever:
16    def __init__(self, pool: asyncpg.Pool, openai: AsyncOpenAI,
17                 semantic_top_k: int = 20, bm25_top_k: int = 20,
18                 rrf_k: int = 60):
19        self.pool = pool
20        self.openai = openai
21        self.semantic_top_k = semantic_top_k
22        self.bm25_top_k = bm25_top_k
23        self.rrf_k = rrf_k
24
25    async def semantic_search(self, query: str) -> list[RetrievedChunk]:
26        resp = await self.openai.embeddings.create(
27            model="text-embedding-3-small", input=query
28        )
29        embedding = resp.data[0].embedding
30        sql = """
31            SELECT chunk_id, content, metadata,
32                   1 - (embedding <=> $1::vector) AS cosine_sim
33            FROM document_chunks
34            ORDER BY embedding <=> $1::vector
35            LIMIT $2
36        """
37        async with self.pool.acquire() as conn:
38            rows = await conn.fetch(sql, str(embedding), self.semantic_top_k)
39        return [
40            RetrievedChunk(
41                chunk_id=r["chunk_id"], text=r["content"],
42                metadata=dict(r["metadata"]) if r["metadata"] else {},
43                score=float(r["cosine_sim"]), rank=i + 1,
44                source="semantic"
45            ) for i, r in enumerate(rows)
46        ]
47
48    async def bm25_search(self, query: str) -> list[RetrievedChunk]:
49        sql = """
50            SELECT chunk_id, content, metadata,
51                   ts_rank(fts_vector, plainto_tsquery('english', $1)) AS rank_score
52            FROM document_chunks
53            WHERE fts_vector @@ plainto_tsquery('english', $1)
54            ORDER BY rank_score DESC
55            LIMIT $2
56        """
57        async with self.pool.acquire() as conn:
58            rows = await conn.fetch(sql, query, self.bm25_top_k)
59        return [
60            RetrievedChunk(
61                chunk_id=r["chunk_id"], text=r["content"],
62                metadata=dict(r["metadata"]) if r["metadata"] else {},
63                score=float(r["rank_score"]), rank=i + 1,
64                source="bm25"
65            ) for i, r in enumerate(rows)
66        ]
67
68    async def retrieve(self, query: str,
69                       top_n: int = 10) -> list[RetrievedChunk]:
70        sem_results, bm25_results = await asyncio.gather(
71            self.semantic_search(query),
72            self.bm25_search(query)
73        )
74        return self._reciprocal_rank_fusion(
75            [sem_results, bm25_results], top_n
76        )
77
78    def _reciprocal_rank_fusion(
79        self, result_lists: list[list[RetrievedChunk]], top_n: int
80    ) -> list[RetrievedChunk]:
81        fused: dict[str, RetrievedChunk] = {}
82        for results in result_lists:
83            for chunk in results:
84                cid = chunk.chunk_id
85                rrf_score = 1.0 / (self.rrf_k + chunk.rank)
86                if cid in fused:
87                    fused[cid].score += rrf_score
88                    fused[cid].source = "hybrid"
89                else:
90                    chunk.score = rrf_score
91                    fused[cid] = chunk
92        ranked = sorted(fused.values(), key=lambda c: c.score, reverse=True)
93        for i, chunk in enumerate(ranked):
94            chunk.rank = i + 1
95        return ranked[:top_n]

• Imports and dataclass: asyncio enables concurrent execution, dataclass gives structured return types, AsyncOpenAI provides non-blocking embedding calls, and asyncpg offers async PostgreSQL access. RetrievedChunk carries the chunk identifier, raw text, metadata dictionary, numeric score, rank position, and a source label indicating semantic, BM25, or hybrid origin.
• HybridRetriever.__init__(): stores the connection pool, OpenAI client, per-source top-k limits, and the RRF smoothing constant rrf_k. Setting semantic_top_k and bm25_top_k to 20 each means the fusion stage considers up to 40 candidate chunks before selecting the final top-N.
• retrieve(): launches both search methods concurrently via asyncio.gather(), collecting their results into two lists, then delegates to the private _reciprocal_rank_fusion() method to merge and deduplicate.
• _reciprocal_rank_fusion(): accumulates RRF scores per chunk, deduplicates by chunk_id, and returns the top-N chunks sorted by fused score. For each chunk in each source list it computes the contribution as 1.0 / (rrf_k + rank). With the default rrf_k=60, a rank-1 chunk contributes 1/61 ≈ 0.0164 and a rank-20 chunk contributes 1/80 = 0.0125 — gentle decay so mid-ranked chunks still meaningfully shift the merged ordering. Chunks appearing in both lists get their scores summed and source flipped to "hybrid". The method is a synchronous helper because it performs no I/O — only in-memory dictionary operations over at most semantic_top_k + bm25_top_k entries.

PostgreSQL schema for dual-index storage

Both retrieval paths depend on the document_chunks table having the correct column types and indexes. The embedding column stores pgvector vectors for cosine search, while fts_vector is a stored generated column that PostgreSQL automatically maintains from the content text. The following SQL creates the table with both a HNSW index for approximate nearest neighbor search and a GIN index for full-text search, enabling the HybridRetriever queries to execute efficiently at scale.

1SCHEMA_SQL = """
2CREATE EXTENSION IF NOT EXISTS vector;
3
4CREATE TABLE IF NOT EXISTS document_chunks (
5    chunk_id    TEXT PRIMARY KEY,
6    document_id TEXT NOT NULL,
7    content     TEXT NOT NULL,
8    metadata    JSONB DEFAULT '{}'::jsonb,
9    embedding   vector(1536),
10    fts_vector  tsvector GENERATED ALWAYS AS (
11                    to_tsvector('english', content)
12                ) STORED,
13    created_at  TIMESTAMPTZ DEFAULT now()
14);
15
16CREATE INDEX IF NOT EXISTS idx_chunks_embedding
17    ON document_chunks USING hnsw (embedding vector_cosine_ops)
18    WITH (m = 16, ef_construction = 64);
19
20CREATE INDEX IF NOT EXISTS idx_chunks_fts
21    ON document_chunks USING gin (fts_vector);
22
23CREATE INDEX IF NOT EXISTS idx_chunks_document
24    ON document_chunks (document_id);
25"""

Do's and Don'ts

Do's

1. ✓Do run semantic_search() and bm25_search() concurrently via asyncio.gather() — the two database queries are fully independent, so parallel execution caps wall-clock retrieval time at max(semantic, BM25) latency rather than their sum, keeping the retrieval stage under 20 ms even when both the HNSW and GIN indexes are cold.
2. ✓Do deduplicate merged results by chunk_id before emitting the ranked list — a chunk that appears in both the pgvector and BM25 result sets accumulates RRF score from two rank positions (Σ 1/(60+r)); without deduplication the downstream reranker receives duplicate context and the fused ranking over-weights that chunk relative to genuinely distinct evidence.
3. ✓Do cast the embedding list to ::vector in the pgvector SQL parameter — asyncpg serializes a Python list as text, and the <=> cosine distance operator requires a properly typed vector argument; the $1::vector cast in the semantic_search() SQL is what converts the stringified embedding correctly, and stripping it causes a PostgreSQL type mismatch at runtime.

Don'ts

1. ✗Don't rely on semantic embeddings alone for precision queries like ERR_CONN_REFUSED troubleshooting — text-embedding-3-small smears exact error codes and version strings into semantically adjacent concepts, returning vaguely related networking prose instead of the exact runbook; the BM25 path's ts_rank over plainto_tsquery is the component that anchors retrieval on literal tokens when vocabulary precision matters.
2. ✗Don't substitute to_tsquery for plainto_tsquery when building the BM25 predicate from raw user input — to_tsquery requires properly formatted tsquery syntax (e.g., 'connection & refused') and raises a PostgreSQL error on unstructured natural-language queries; plainto_tsquery('english', $1) tokenizes and stems arbitrary user text safely and is the correct function for the fts_vector @@ ... predicate in bm25_search().
3. ✗Don't omit the GIN index on fts_vector or the HNSW index on embedding — without GIN, @@ plainto_tsquery(...) falls back to a sequential scan instead of the inverted index, turning a 2–8 ms BM25 query into a full-table scan; without HNSW, pgvector performs exact nearest-neighbor search instead of approximate, adding seconds of latency on any table past 100K rows and negating the sub-20 ms budget that asyncio.gather() is designed to preserve.