Build an embedding benchmarking framework

Introduction

When you pick an embedding provider from vendor leaderboards alone, the numbers rarely survive contact with your own corpus, and the gap shows up as silent quality regressions and runaway costs once the pipeline reaches production. Teams that skip on-domain benchmarking end up either overpaying for a premium provider they don't need or quietly shipping a cheaper one that hurts answer quality across every downstream RAG response. This lesson walks through a reusable benchmarking framework that runs any embedding provider against your own queries, documents, and relevance judgments, then reports MRR, NDCG, and recall@k alongside latency and cost. By the end you will be able to score multiple providers head-to-head on domain data and defend a selection decision with numbers instead of vendor marketing.

Key Terminology

• MRR (Mean Reciprocal Rank): average of 1/rank for the first relevant document returned per query; rewards models that surface a correct hit early.
• NDCG@k (Normalized Discounted Cumulative Gain): position-weighted retrieval score in the top-k results, normalized against the ideal ranking so values are comparable across queries.
• Recall@k: fraction of relevant documents that appear within the top-k results, indicating how completely the retriever covers the relevant set.
• Relevance judgments: per-query sets of document indices marked as correct answers; the ground truth the framework grades each provider against.
• Provider-agnostic embed_fn: a callable that maps a list of texts to a list of embedding vectors, allowing the harness to swap providers without touching evaluation logic.

Concepts

The framework separates three concerns so each can evolve independently. First, the dataset layer holds queries, documents, and relevance judgments — domain data you control and reuse across every provider run. Second, the provider adapter is a single embed_fn callable; any new model is onboarded by writing one function, never by editing the scorer. Third, the evaluation layer computes complementary metrics: MRR captures how often the first hit is right, NDCG rewards correct ranking in the top window, and recall@k measures coverage. Running all three together prevents a provider from gaming a single number while losing on the others. Latency and cost are tracked alongside quality, because a provider that wins on NDCG by 1% but costs 5× more or doubles p99 latency is rarely the right production choice.

Code Walkthrough

Now that you have seen the concepts above, the walkthrough below turns them into working code.

Building the Benchmarking Harness

The benchmarking harness accepts a set of queries, a document corpus, and relevance judgments, then evaluates any embedding provider against three complementary retrieval metrics. Its provider-agnostic design means adding a new embedding provider requires only implementing a single callable that maps text lists to embedding lists, with no changes to the evaluation logic itself. This separation of concerns is critical in the fast-moving embedding landscape, where new providers appear quarterly and existing providers release updated models that need re-evaluation against your domain-specific data.

1import numpy as np
2from dataclasses import dataclass
3from typing import Callable
4import time
5
6@dataclass
7class BenchmarkResult:
8    provider: str
9    mrr: float
10    ndcg_at_10: float
11    recall_at_5: float
12    recall_at_10: float
13    avg_latency_ms: float
14    total_cost_usd: float
15
16class EmbeddingBenchmark:
17    def __init__(self, queries: list[str], documents: list[str], relevant: list[set[int]]):
18        self.queries = queries
19        self.documents = documents
20        self.relevant = relevant
21
22    def benchmark_provider(
23        self,
24        provider_name: str,
25        embed_fn: Callable[[list[str]], list[list[float]]],
26        cost_per_token: float = 0.0,
27    ) -> BenchmarkResult:
28        start = time.perf_counter()
29        doc_embeddings = embed_fn(self.documents)
30        query_embeddings = embed_fn(self.queries)
31        total_time = time.perf_counter() - start
32
33        rankings = []
34        for q_idx in range(len(self.queries)):
35            q_emb = np.array(query_embeddings[q_idx])
36            scores = [float(np.dot(q_emb, np.array(d))) for d in doc_embeddings]
37            ranked = np.argsort(scores)[::-1].tolist()
38            rankings.append(ranked)
39
40        return BenchmarkResult(
41            provider=provider_name,
42            mrr=self._compute_mrr(rankings),
43            ndcg_at_10=self._compute_ndcg(rankings, k=10),
44            recall_at_5=self._compute_recall(rankings, k=5),
45            recall_at_10=self._compute_recall(rankings, k=10),
46            avg_latency_ms=(total_time / (len(self.queries) + len(self.documents))) * 1000,
47            total_cost_usd=0.0,
48        )
49
50    def _compute_mrr(self, rankings: list[list[int]]) -> float:
51        rrs = []
52        for ranked, rel in zip(rankings, self.relevant):
53            for rank, doc_idx in enumerate(ranked, 1):
54                if doc_idx in rel:
55                    rrs.append(1.0 / rank)
56                    break
57            else:
58                rrs.append(0.0)
59        return float(np.mean(rrs))
60
61    def _compute_ndcg(self, rankings: list[list[int]], k: int) -> float:
62        ndcgs = []
63        for ranked, rel in zip(rankings, self.relevant):
64            dcg = sum(
65                1.0 / np.log2(rank + 2)
66                for rank, doc_idx in enumerate(ranked[:k])
67                if doc_idx in rel
68            )
69            ideal = sum(1.0 / np.log2(i + 2) for i in range(min(len(rel), k)))
70            ndcgs.append(dcg / max(ideal, 1e-9))
71        return float(np.mean(ndcgs))
72
73    def _compute_recall(self, rankings: list[list[int]], k: int) -> float:
74        recalls = []
75        for ranked, rel in zip(rankings, self.relevant):
76            found = len(set(ranked[:k]) & rel)
77            recalls.append(found / max(len(rel), 1))
78        return float(np.mean(recalls))

Measuring Latency and Throughput

Beyond retrieval quality, embedding generation latency directly impacts pipeline throughput and user-facing search response times. The measure_latency function below profiles each provider at multiple batch sizes (1, 10, 50, 100 texts), running three iterations per batch size to compute stable averages. This reveals the optimal batch configuration per provider, since some providers achieve significantly better throughput at larger batch sizes due to amortized network overhead, while others show diminishing returns beyond a certain batch size.

1def measure_latency(
2    embed_fn: Callable,
3    texts: list[str],
4    batch_sizes: list[int] = [1, 10, 50, 100],
5) -> list[dict]:
6    results = []
7    for batch_size in batch_sizes:
8        batch = texts[:batch_size]
9        times = []
10        for _ in range(3):
11            start = time.perf_counter()
12            embed_fn(batch)
13            elapsed = time.perf_counter() - start
14            times.append(elapsed)
15        avg = np.mean(times)
16        results.append({
17            "batch_size": batch_size,
18            "avg_seconds": round(avg, 3),
19            "texts_per_second": round(batch_size / avg, 1),
20        })
21    return results

This framework produces the complete data needed for provider selection: quality metrics across three dimensions (MRR, NDCG, recall), latency profiles at different batch sizes, and cost projections based on actual token consumption.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Score every provider on the same queries, documents, and relevance judgments so results are directly comparable.
2. ✓Report MRR, NDCG@10, and recall@k together — never pick a provider on a single metric.
3. ✓Measure latency at multiple batch sizes (1, 10, 50, 100) so the throughput profile reflects how the provider will actually be called in production.

Don'ts

1. ✗Don't reuse a vendor's published benchmark numbers as a stand-in for evaluation on your own domain data.
2. ✗Don't compare providers using different document sets, query sets, or relevance judgments — any quality delta will be noise.
3. ✗Don't ignore latency and cost; a marginally higher NDCG rarely justifies a 5× cost or a doubled p99 latency in production.