Compare retrieval quality across embedding models with Cohere Rerank

Introduction

Engineers often swap embedding models mid-project — switching from a smaller open-source model to a larger provider API — only to discover later that retrieval quality quietly degraded. Without a systematic comparison framework, there is no way to detect which model and reranker combination actually serves your RAG pipeline best. By the end of this lesson, you'll be able to instrument a parallel evaluation runner that benchmarks multiple embedding-model and Cohere Rerank combinations against the same query set, producing precision, recall, and NDCG@10 metrics stored in structured Pydantic schemas and exposed as Prometheus gauges.

Key Terminology

• Retrieval Precision — the fraction of retrieved (or reranked) documents that are relevant to a query; computed per query inside evaluate_combination and stored in ComparisonResult.retrieval_precision as the primary signal for model comparison.
• NDCG@10 — Normalized Discounted Cumulative Gain at cutoff 10, a ranking metric that rewards relevant documents appearing earlier in the result list rather than later; captured in ComparisonResult.ndcg_at_10 and populated by downstream RAGAS evaluation steps.
• ModelCombination — the unit of comparison in an evaluation sweep: a Pydantic schema that pairs an EmbeddingModelConfig with an optional RerankConfig under a stable combination_id used to label Prometheus metrics and join ComparisonResult records across queries.
• Cohere Rerank — a second-pass relevance scoring API (rerank-v3.5) that re-orders an initial candidate document list by query relevance; toggled via RerankConfig.enabled so the same embedding model can be benchmarked with and without reranking in a single sweep.
• asyncio.to_thread — a Python stdlib utility that runs a synchronous callable in a background thread pool; used here to call Cohere's blocking rerank SDK from within evaluate_combination without stalling the async event loop.
• Prometheus Gauge and Histogram — observability instruments emitted by ParallelEvaluationRunner: COMPARISON_PRECISION is a Gauge that holds the current precision score per combination_id and segment, while COMPARISON_LATENCY is a Histogram that records the evaluation latency distribution across sub-second-to-10s buckets.

Concepts

Two-Stage Retrieval: Vector Search Followed by Reranking

Most RAG pipelines perform a single retrieval pass — a vector similarity search returning the top-k candidates. Similarity rank and relevance rank diverge more than engineers expect: a document that scores highly on cosine distance may still be a poor answer to the query because embedding models compress semantics in ways that do not fully capture fine-grained relevance distinctions. Reranking adds a second, more expensive pass that re-scores those candidates using a dedicated relevance model. Cohere Rerank (rerank-v3.5) compares each candidate document against the query and returns a relevance-ordered list, which can recover highly relevant documents that the embedding model ranked too low.

The RerankConfig.enabled flag is what makes this comparison systematic. Setting it to False runs the embedding model alone and records a baseline precision score; setting it to True applies reranking to the same retrieved document list before computing precision. Because both variants share the same EmbeddingModelConfig and execute within one sweep, differences in the resulting ComparisonResult records are attributable entirely to the reranking step, not to any change in the retrieval corpus or query set.

Schema-Driven Comparison: Why combination_id Is the Load-Bearing Field

Without a stable identifier linking model configuration to metrics, a multi-model benchmark produces numbers with no provenance — you cannot tell which precision score belongs to which model version a week after the run. ModelCombination.combination_id is that anchor: it appears in ComparisonResult, labels the COMPARISON_PRECISION Prometheus gauge, and groups evaluation records so dashboards and downstream RAGAS steps can join results by model configuration rather than by run order.

The schema hierarchy enforces lineage at every layer. EmbeddingModelConfig.dimension matters specifically because embedding models with different output dimensions require separate vector collections; comparing a 768-dim model against a 1536-dim model in the same collection would corrupt retrieval silently. By capturing dimension inside the schema, every ComparisonResult carries enough metadata to reconstruct which vector index was queried (see Code Walkthrough for how these configs are declared and threaded through the runner).

Bridging Synchronous SDKs in an Async Evaluation Runner

ParallelEvaluationRunner.evaluate_combination is an async method that needs to call Cohere's Python SDK — which is synchronous and blocking. Calling it directly inside the coroutine would freeze the event loop for the entire duration of the HTTP round-trip, serializing every concurrent evaluation and defeating the point of an async runner. asyncio.to_thread offloads the blocking call to a thread-pool worker, returning control to the event loop immediately so other combinations can make progress in parallel.

This pattern — wrapping synchronous vendor SDK calls with asyncio.to_thread — is the standard bridge whenever an async pipeline depends on a sync library. Understanding it matters beyond this lesson: any additional synchronous reranking or embedding provider you add to the same runner must follow the same wrapping pattern, or it becomes an unintentional bottleneck that skews the latency recorded in ComparisonResult.latency_ms and the COMPARISON_LATENCY histogram (see Code Walkthrough).

Code Walkthrough

Now that you have the core metrics vocabulary from the Concepts section, the implementation follows two layers: a Pydantic schema layer that captures per-query results, and a ParallelEvaluationRunner that drives Cohere Rerank and emits Prometheus observability.

The schema layer defines four models. EmbeddingModelConfig records the provider, version, and output dimension — important because dimension mismatches between models require separate vector collections. RerankConfig defaults to Cohere's rerank-v3.5 with a configurable top_n cutoff, and its enabled flag lets you compare the same embedding model with and without reranking in a single sweep. ModelCombination pairs an embedding config with an optional rerank config under a unique combination_id used to track results. ComparisonResult captures the per-query metrics — retrieval_precision, retrieval_recall, ndcg_at_10, and latency_ms — that downstream dashboards read to detect quality regressions.

Code snippetpython
1from datetime import datetime
2from typing import Optional
3from pydantic import BaseModel, Field
4
5class EmbeddingModelConfig(BaseModel):
6    model_name: str
7    model_version: str
8    provider: str
9    dimension: int
10    metadata: dict = Field(default_factory=dict)
11
12class RerankConfig(BaseModel):
13    enabled: bool = False
14    model_name: str = "rerank-v3.5"
15    provider: str = "cohere"
16    top_n: int = 10
17
18class ModelCombination(BaseModel):
19    embedding: EmbeddingModelConfig
20    rerank: Optional[RerankConfig] = None
21    combination_id: str
22
23class ComparisonResult(BaseModel):
24    combination_id: str
25    query_id: str
26    retrieval_precision: float
27    retrieval_recall: float
28    ndcg_at_10: float
29    rerank_applied: bool
30    latency_ms: int
31    evaluated_at: datetime = Field(default_factory=datetime.utcnow)

The runner wraps a cohere.Client and exposes two Prometheus instruments: a Gauge tracking per-combination precision segmented by combination_id and segment, and a Histogram measuring evaluation latency with sub-second-to-10s buckets. When rerank.enabled is True, the runner calls cohere.rerank via asyncio.to_thread — because Cohere's SDK is synchronous — to avoid blocking the event loop, then re-orders the retrieved documents by relevance rank before computing precision. The result is recorded in Prometheus and returned as a ComparisonResult for storage and downstream RAGAS evaluation.

Code snippetpython
1from models import ModelCombination, ComparisonResult
2import cohere
3import asyncio
4import time
5from prometheus_client import Gauge, Histogram
6
7COMPARISON_PRECISION = Gauge(
8    "embedding_comparison_precision",
9    "Retrieval precision by model combination",
10    ["combination_id", "segment"],
11)
12COMPARISON_LATENCY = Histogram(
13    "embedding_comparison_latency_seconds",
14    "Evaluation latency per combination",
15    ["combination_id"],
16    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0],
17)
18
19class ParallelEvaluationRunner:
20    def __init__(self, cohere_client: cohere.Client):
21        self._co = cohere_client
22
23    async def evaluate_combination(
24        self,
25        combination: ModelCombination,
26        query: str,
27        retrieved_docs: list[str],
28        segment: str,
29    ) -> ComparisonResult:
30        start = time.monotonic()
31        docs_to_score = retrieved_docs
32
33        if combination.rerank and combination.rerank.enabled:
34            response = await asyncio.to_thread(
35                self._co.rerank,
36                model=combination.rerank.model_name,
37                query=query,
38                documents=retrieved_docs,
39                top_n=combination.rerank.top_n,
40            )
41            docs_to_score = [
42                retrieved_docs[r.index] for r in response.results
43            ]
44
45        relevant = sum(1 for d in docs_to_score if query.lower() in d.lower())
46        precision = relevant / len(docs_to_score) if docs_to_score else 0.0
47        latency_ms = int((time.monotonic() - start) * 1000)
48
49        COMPARISON_PRECISION.labels(
50            combination_id=combination.combination_id,
51            segment=segment,
52        ).set(precision)
53        COMPARISON_LATENCY.labels(
54            combination_id=combination.combination_id,
55        ).observe(latency_ms / 1000)
56
57        return ComparisonResult(
58            combination_id=combination.combination_id,
59            query_id=query[:32],
60            retrieval_precision=precision,
61            retrieval_recall=0.0,
62            ndcg_at_10=0.0,
63            rerank_applied=bool(
64                combination.rerank and combination.rerank.enabled
65            ),
66            latency_ms=latency_ms,
67        )

Verify by instantiating a ParallelEvaluationRunner with a real Cohere client and a ModelCombination where rerank.enabled=True, then calling evaluate_combination with a sample query and document list — result.rerank_applied should be True, result.latency_ms should be non-zero, and the embedding_comparison_precision Prometheus gauge should register a value for the combination's labels.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do wrap Cohere's synchronous rerank call in asyncio.to_thread — Cohere's SDK is blocking, so calling it directly inside an async method stalls the entire event loop; asyncio.to_thread offloads the call to a thread pool and keeps the runner non-blocking under concurrent evaluation sweeps.
2. ✓Do use RerankConfig.enabled as the toggle when benchmarking the same embedding model with and without reranking — setting enabled=False within the same ModelCombination sweep lets ParallelEvaluationRunner.evaluate_combination skip the cohere.rerank call and score the original retrieval order, giving you a clean apples-to-apples comparison of reranking's marginal NDCG@10 lift.
3. ✓Do label COMPARISON_PRECISION and COMPARISON_LATENCY with combination_id — every ModelCombination carries a unique combination_id, and without it as a Prometheus label, gauge updates from different embedding-model and reranker pairs overwrite each other, making it impossible to detect which specific combination caused a precision regression in your dashboards.

Don'ts

1. ✗Don't assume a single vector collection works across embedding models with different dimension values — EmbeddingModelConfig.dimension exists precisely because switching from a smaller open-source model to a larger provider API often changes the output dimensionality, requiring a separate collection; reusing the same index silently corrupts retrieval results before any metric is computed.
2. ✗Don't treat retrieval_recall and ndcg_at_10 as non-zero just because rerank_applied is True — the ParallelEvaluationRunner populates retrieval_precision from the reranked docs_to_score but stubs retrieval_recall=0.0 and ndcg_at_10=0.0; connecting those fields to your ground-truth relevance labels is a required downstream step before using ComparisonResult for production quality-degradation alerting.
3. ✗Don't call cohere.rerank without binding top_n from RerankConfig — hard-coding a constant top_n in the call site bypasses the config and breaks the with/without-reranking comparison, since the number of documents re-ordered before precision is computed must be consistent across all ModelCombination entries in the sweep to produce comparable retrieval_precision values.