Implement embedding quality checks

Vector embeddings sit at the foundation of every retrieval-augmented generation (RAG) pipeline, semantic search index, and similarity-based recommendation system. Unlike traditional tabular data where a malformed row triggers an obvious schema violation, embedding corruption is insidious — a silently truncated 1536-dimension vector still looks like a list of floats, yet it poisons every downstream cosine-similarity calculation. This section equips you to build systematic quality checks that catch three categories of embedding defects before they reach production: dimensional consistency (every vector matches the expected shape and norm range), completeness (no records are missing their embedding or associated metadata), and freshness (embeddings reflect the most recent version of their source documents rather than stale content).

The quality checks you will build here integrate naturally with the Great Expectations validation framework and Pydantic models for record-level schema enforcement. Rather than treating embedding validation as an afterthought, you will learn to embed these checks directly into your data pipeline as automated quality gates — hard stops that prevent degraded data from flowing into your vector store or fine-tuning dataset.

Introduction

When you ship a RAG service to production, embeddings are the data your users implicitly trust — and the easiest data in your stack to corrupt without anyone noticing. A truncated vector, a mismatched model version, or a stale embedding all return valid floats and zero schema errors, yet each silently degrades retrieval quality. The consequence is a search product that looks healthy on every dashboard while quietly returning the wrong documents. By the end of this lesson you will be able to wire a three-layer quality gate — geometric, completeness, and freshness — into your embedding pipeline so corrupted, missing, or stale vectors are caught before they ever reach your vector store.

Key Terminology

• Embedding — a fixed-length vector representation of a document or query that places semantically similar inputs near each other in vector space; this is the unit of data every check in this lesson protects.
• Geometric validation — checks that operate on a vector's mathematical properties (dimensionality, L2 norm, all-zero detection) and catch corruption that schema validators miss because the row still type-checks as a list of floats.
• Completeness validation — cross-references the source document registry against the embedding store to surface missing embeddings (documents never embedded) and orphaned embeddings (vectors whose source document was deleted).
• Freshness — the property that an embedding still reflects its source document's current content; freshness decays silently whenever a document is updated without re-running the embedding pipeline.
• Quality gate — a hard stop in the pipeline that refuses to upsert a batch of embeddings into the vector store unless every geometric, completeness, and freshness check passes.

Concepts

Connecting to Data Quality Metrics and Anomaly Detection

The quality gate function produces structured output that feeds directly into data quality monitoring systems. In production, you should track these metrics over time to detect data drift — the gradual degradation of embedding quality that occurs as source data evolves, models are updated, or pipeline components silently fail. Key metrics to monitor include:

• Dimension failure rate: Should always be exactly 0.0 in a healthy pipeline. Any non-zero value indicates a model version mismatch or serialization bug that requires immediate investigation.
• Norm outlier rate: Track the percentage of vectors with norms outside your configured bounds. A sudden spike often correlates with an upstream model change or API version rollover.
• Zero vector rate: Monitor this as a canary metric. Even a single zero vector in a production batch usually signals an API failure that should trigger an alert.
• Average freshness score: This metric should hover near 1.0 for active pipelines. A gradual decline toward 0.0 indicates that your re-embedding schedule is not keeping pace with document updates — a form of data drift that degrades retrieval quality without triggering hard failures.
• Missing embedding count: Should trend toward zero as your pipeline processes its backlog. A sustained non-zero count suggests that your ingestion pipeline is dropping records.

When combined with Great Expectations, these metrics can be expressed as expectation suites that run automatically on each pipeline execution. For example, you can define a Great Expectations checkpoint that asserts the dimension failure rate is exactly zero, the norm outlier rate is below 0.1%, and the average freshness score exceeds 0.9. This transforms your ad-hoc quality checks into a formalized data contract — a documented, versioned agreement between the embedding generation service and its downstream consumers about the minimum quality standards that the data must meet.

The combination of Pydantic validation for schema enforcement (catching structural defects at the record level), the EmbeddingQualityChecker for geometric validation (catching mathematical defects at the vector level), and the CompletenessChecker for registry-level validation (catching systemic gaps in coverage and freshness) gives you a three-layer defense that catches embedding quality issues at every granularity. Each layer operates independently, reports its results in a structured format suitable for automated monitoring, and integrates cleanly into pipeline orchestrators as a blocking quality gate that prevents degraded data from reaching your vector store or training dataset.

Code Walkthrough

Now that you understand the three validation layers — geometric, completeness, and freshness — the following implementation shows how each maps to a concrete method on a single EmbeddingQualityChecker class.

Geometric checks operate on vector math: dimensionality (every vector must match the model's expected output size), L2 norm (vectors outside a narrow band around 1.0 indicate a model mismatch or serialization corruption), and zero-vector detection (a common artifact of silent API failures that return a default value). The validate method composes all three checks into a single QualityReport whose passed field acts as the quality gate signal — any non-empty failure list flips it to False and blocks the batch from reaching the vector store.

Code snippetpython
1import numpy as np
2from dataclasses import dataclass, field
3
4@dataclass
5class QualityReport:
6    total_vectors: int = 0
7    dimension_failures: list[int] = field(default_factory=list)
8    norm_outliers: list[int] = field(default_factory=list)
9    zero_vectors: list[int] = field(default_factory=list)
10    passed: bool = False
11
12class EmbeddingQualityChecker:
13    def __init__(
14        self,
15        expected_dim: int = 1536,
16        norm_min: float = 0.8,
17        norm_max: float = 1.2,
18    ):
19        self.expected_dim = expected_dim
20        self.norm_min = norm_min
21        self.norm_max = norm_max
22
23    def check_dimensions(self, embeddings: list[list[float]]) -> list[int]:
24        return [i for i, vec in enumerate(embeddings) if len(vec) != self.expected_dim]
25
26    def check_norms(self, embeddings: list[list[float]]) -> list[int]:
27        outliers = []
28        for i, vec in enumerate(embeddings):
29            norm = float(np.linalg.norm(vec))
30            if norm < self.norm_min or norm > self.norm_max:
31                outliers.append(i)
32        return outliers
33
34    def check_zero_vectors(self, embeddings: list[list[float]]) -> list[int]:
35        return [i for i, vec in enumerate(embeddings) if all(v == 0.0 for v in vec)]
36
37    def validate(self, embeddings: list[list[float]]) -> QualityReport:
38        report = QualityReport(total_vectors=len(embeddings))
39        report.dimension_failures = self.check_dimensions(embeddings)
40        report.norm_outliers = self.check_norms(embeddings)
41        report.zero_vectors = self.check_zero_vectors(embeddings)
42        report.passed = (
43            len(report.dimension_failures) == 0
44            and len(report.norm_outliers) == 0
45            and len(report.zero_vectors) == 0
46        )
47        return report

The three index lists on QualityReport tell downstream code exactly which vectors failed so they can be routed to a quarantine queue for re-embedding rather than silently dropped. Once you have a report, you can derive the monitoring metrics described in the Concepts section — dimension failure rate, norm outlier rate, and zero vector rate — directly from its fields:

Code snippetpython
1def summarize(report: QualityReport) -> dict:
2    n = report.total_vectors or 1  # guard against empty batch
3    return {
4        "dimension_failure_rate": len(report.dimension_failures) / n,
5        "norm_outlier_rate": len(report.norm_outliers) / n,
6        "zero_vector_rate": len(report.zero_vectors) / n,
7        "passed": report.passed,
8    }

Feeding summarize(report) into your monitoring system on every pipeline run lets you track each rate over time. A dimension failure rate above 0.0 demands immediate investigation; a rising norm outlier rate often signals an upstream model change; a non-zero zero vector rate should trigger an alert on the same run it appears.

You've completed this when EmbeddingQualityChecker().validate(batch).passed returns True for a representative sample of freshly generated embeddings and all three failure lists are empty.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do initialize EmbeddingQualityChecker with the exact expected_dim your embedding model outputs — passing 1536 for a text-embedding-3-small model versus 3072 for text-embedding-3-large are not interchangeable, and a wrong dimension silently passes every vector through check_dimensions even when the model was swapped mid-pipeline.
2. ✓Do route index positions from QualityReport.dimension_failures, norm_outliers, and zero_vectors to a quarantine queue for re-embedding — these lists identify exactly which vectors failed so they can be recovered rather than dropped, and discarding them silently shrinks your vector store without any schema error or dashboard alert.
3. ✓Do emit the rates from summarize(report) into your monitoring system on every pipeline run — a dimension failure rate above 0.0 signals an immediate model or serialization break, a rising norm_outlier_rate often indicates an upstream model change, and any non-zero zero_vector_rate means silent API failures are producing default-value vectors that will corrupt retrieval.

Don'ts

1. ✗Don't rely on report.passed alone without inspecting the three index lists — passed is a gate signal that collapses all failures into a boolean, but the downstream quarantine step needs the specific indices from dimension_failures, norm_outliers, and zero_vectors to re-embed the right vectors; ignoring the lists means corrupted vectors are either silently re-ingested or silently lost.
2. ✗Don't set norm_min and norm_max so wide that they stop catching model mismatches — the [0.8, 1.2] band around 1.0 is deliberately narrow because properly normalized embeddings cluster tightly near 1.0, and widening the band to avoid noisy alerts defeats check_norms's ability to catch serialization corruption or a switched embedding model that outputs at a different scale.
3. ✗Don't skip the zero-vector check on the assumption that your embedding API never fails silently — check_zero_vectors exists precisely because a silent API failure (network timeout, rate-limit swallowed by a retry wrapper) commonly returns a default all-zeros vector that passes dimension and norm checks yet destroys cosine similarity for every query it matches.