Build zero-downtime reindexing for embedding model upgrades

Introduction

When you upgrade from text-embedding-004 to text-embedding-005, every stored vector becomes incompatible with query vectors from the new model — recall collapses overnight and your search results turn into noise. Teams that drop the old column, regenerate every embedding, and rebuild the index pay for it with hours of downtime during which no query can be served. By the end of this lesson you'll be able to design a blue-green reindexing flow that runs both models in parallel, validates the new index against recall and latency budgets, and cuts traffic over with an instant rollback path.

Key Terminology

• Blue-green reindex — a migration pattern that keeps the old (blue) embedding column live while a new (green) column is populated in parallel; query traffic flips between them with a metadata change, giving zero-downtime cutover and instant rollback.
• Concurrent index build — CREATE INDEX CONCURRENTLY in Postgres / pgvector builds the green HNSW index without blocking reads or writes against the blue column, so production keeps serving traffic for the entire build window.
• Recall delta — the difference between the green index's recall and the blue index's recall on the same query set; a tiny negative delta is tolerable, a large one means the new model is degrading user-visible result quality.
• Cutover validation — a gate that benchmarks both columns against the same query vectors and refuses the switch unless recall and p99 latency stay within configured budgets.

Concepts

Blue-green index strategy

Add a new vector column (embedding_v2) alongside the existing embedding column. Each column gets its own HNSW index, both are queryable, and the application chooses between them with a feature flag. The migration is staged — prepare the green column, backfill embeddings in batches, validate, then flip query traffic. Because the green index is built next to the live blue index, the blue index keeps serving traffic the whole time (see Code Walkthrough).

Traffic switching with validation

The cutover itself is just a metadata change — which column name the query layer reads from. The risk is not in the switch; it's in switching to an index that has worse recall or latency than the one it replaces. A validator benchmarks both columns against the same query set, compares recall and p99 latency against the blue baseline, and refuses the switch unless the deltas fit inside configured budgets. Rollback is the same metadata flip in reverse (see Code Walkthrough).

Code Walkthrough

Now that you have the blue-green strategy and the cutover-validation gate from the previous section, the two snippets below make them concrete. The first shows BlueGreenMigrator managing the lifecycle of the green column — adding it, building its HNSW index concurrently, and reporting backfill progress. The second shows TrafficSwitcher enforcing the cutover gate by benchmarking both columns and refusing the flip when budgets aren't met.

Code snippetpython
1class BlueGreenMigrator:
2    def __init__(
3        self,
4        db_pool: asyncpg.Pool,
5        old_column: str = "embedding",
6        new_column: str = "embedding_v2",
7        new_dimensions: int = 3072,
8    ):
9        self.db_pool = db_pool
10        self.old_col = old_column
11        self.new_col = new_column
12        self.new_dims = new_dimensions
13
14    async def prepare_green(self) -> None:
15        async with self.db_pool.acquire() as conn:
16            await conn.execute(f"""
17                ALTER TABLE embeddings
18                ADD COLUMN IF NOT EXISTS {self.new_col}
19                vector({self.new_dims})
20            """)
21            await conn.execute(f"""
22                CREATE INDEX CONCURRENTLY
23                IF NOT EXISTS idx_embeddings_green
24                ON embeddings
25                USING hnsw ({self.new_col} vector_cosine_ops)
26                WITH (m = 16, ef_construction = 256)
27            """)
28
29    async def get_migration_progress(self) -> dict:
30        async with self.db_pool.acquire() as conn:
31            row = await conn.fetchrow(f"""
32                SELECT COUNT(*) AS total,
33                       COUNT({self.new_col}) AS migrated
34                FROM embeddings
35            """)
36        return {
37            "total": row["total"],
38            "migrated": row["migrated"],
39            "progress_pct": round(
40                row["migrated"] / max(row["total"], 1) * 100, 1
41            ),
42        }

• prepare_green adds the new vector column and builds its HNSW index with CONCURRENTLY — reads and writes against the blue column are not blocked while the green index is being built.
• new_dimensions is configurable because new embedding models often change dimensionality (1536 → 3072 is common); the column type must match the model output exactly or vector_cosine_ops will reject inserts.
• get_migration_progress reports the percentage of rows that have a green embedding. This drives the dashboard that tells you when the backfill is far enough along to attempt validation.

Once the backfill is far enough along, TrafficSwitcher benchmarks the green column against the blue baseline and refuses to flip if the deltas exceed budget.

Code snippetpython
1@dataclass
2class ValidationResult:
3    old_recall: float
4    new_recall: float
5    old_p99_ms: float
6    new_p99_ms: float
7    passed: bool
8    reason: str
9
10class TrafficSwitcher:
11    def __init__(
12        self,
13        db_pool: asyncpg.Pool,
14        benchmark: IndexBenchmark,
15        min_recall_delta: float = -0.01,
16        max_latency_delta_pct: float = 20.0,
17    ):
18        self.db_pool = db_pool
19        self.benchmark = benchmark
20        self.min_recall_delta = min_recall_delta
21        self.max_latency_delta = max_latency_delta_pct
22
23    async def validate(
24        self,
25        query_vectors: list[list[float]],
26        old_column: str,
27        new_column: str,
28    ) -> ValidationResult:
29        old_metrics = await self._benchmark_column(old_column, query_vectors)
30        new_metrics = await self._benchmark_column(new_column, query_vectors)
31        recall_delta = new_metrics["recall"] - old_metrics["recall"]
32        latency_delta_pct = (
33            (new_metrics["p99"] - old_metrics["p99"])
34            / max(old_metrics["p99"], 0.01) * 100
35        )
36        passed = (
37            recall_delta >= self.min_recall_delta
38            and latency_delta_pct <= self.max_latency_delta
39        )
40        reason = "OK" if passed else (
41            f"recall_delta={recall_delta:.3f}, "
42            f"latency_delta_pct={latency_delta_pct:.1f}"
43        )
44        return ValidationResult(
45            old_recall=old_metrics["recall"],
46            new_recall=new_metrics["recall"],
47            old_p99_ms=old_metrics["p99"],
48            new_p99_ms=new_metrics["p99"],
49            passed=passed,
50            reason=reason,
51        )
52
53    async def _benchmark_column(
54        self, column: str, query_vectors: list[list[float]]
55    ) -> dict:
56        latencies, recalls = [], []
57        for qv in query_vectors:
58            exact = await self.benchmark.exact_search(qv)
59            approx, lat = await self.benchmark.approximate_search(qv)
60            latencies.append(lat * 1000)
61            recalls.append(self.benchmark.compute_recall(exact, approx))
62        return {
63            "recall": float(np.mean(recalls)),
64            "p99": float(np.percentile(latencies, 99)),
65        }

• ValidationResult carries both baselines and the verdict, so the cutover decision is fully auditable — you can replay exactly why a flip was refused.
• min_recall_delta=-0.01 tolerates a 1% recall drop; max_latency_delta_pct=20 tolerates a 20% p99 increase. These are the levers — tune them per workload, but keep them explicit.
• _benchmark_column runs the same query vectors through exact search (ground truth) and approximate search (HNSW) on the specified column, returning mean recall and p99 latency.

You'll know it works when validate reports passed=True with a recall delta within budget on a production-shaped query set, and a re-flip back to the blue column restores baseline recall instantly.

Discipline-Specific Application

The blue-green pattern above is the engine; the part that varies by role is which recall and latency budgets matter, which query sets count as "production-shaped", and what "safe to flip" means for the workload your team owns.

Do's and Don'ts

Having just seen how the migrator builds the green index and the switcher gates the cutover, the rules below distil the choices that most often decide whether a zero-downtime reindex stays zero-downtime.

Do's

1. ✓Do build the green index with CONCURRENTLY — the live blue index keeps serving reads and writes while the green index is being built, which is the entire point of zero-downtime cutover.
2. ✓Do validate against a production-shaped query set — benchmark recall and p99 latency on queries that look like real traffic, not synthetic vectors, or the validator will green-light an index that degrades real users.
3. ✓Do keep the blue column for a grace period after the flip — instant rollback only exists while both columns are live; dropping blue too early turns the flip into a one-way door.

Don'ts

1. ✗Don't drop the old column before the grace period ends — if the new model regresses on a traffic slice you didn't benchmark, you need the blue column to flip back to.
2. ✗Don't run the backfill in one giant transaction — batch in chunks (e.g. 1000 rows) so embedding API failures, rate limits, or pod restarts don't roll back hours of work.
3. ✗Don't widen the recall/latency tolerances to make a bad index pass — if the new model can't meet the budget on a representative query set, the fix is a better model or better index parameters, not laxer thresholds.