Detect scope drift with embedding similarity classification

Introduction

When clients request features mid-engagement, teams often lack a systematic way to distinguish work that genuinely falls within agreed scope from work that represents new delivery commitments. Without automated detection, scope creep compounds silently until budget overruns surface too late. This lesson teaches you how to build an embedding-based scope change detector that compares incoming requests against a vector-indexed baseline of your approved scope document — classifying each request as in-scope, adjacent, or out-of-scope so your governance workflow can act before commitments are made.

Key Terminology

• Scope baseline — the vector-indexed representation of an approved scope document, built by chunking the document at \n## section boundaries and storing one pgvector row per section; it serves as the semantic reference against which all incoming requests are compared.
• Section chunking — the preprocessing step in store_baseline that splits a scope document by \n## header markers before embedding, ensuring each vector encodes a coherent topic rather than an entire undifferentiated document.
• Cosine similarity — a measure of angular alignment between two embedding vectors, computed in the query as 1 - (embedding <=> $2::vector); scores range from 0 (semantically unrelated) to 1 (semantically identical) and drive the three-tier classification logic.
• pgvector — a PostgreSQL extension that stores dense float vectors and supports efficient nearest-neighbor queries via the <=> cosine distance operator, used here to retrieve the closest baseline sections to an incoming request.
• Scope classification — the three-tier label (in_scope, adjacent, out_of_scope) assigned by check_scope_change based on the maximum cosine similarity to any baseline section; thresholds are THRESHOLDS["in_scope"] = 0.85 and THRESHOLDS["adjacent"] = 0.65.
• Re-baselining — the operation of atomically replacing all stored scope vectors for an engagement by deleting existing rows and re-embedding the amended scope document; store_baseline handles this so approved scope amendments are reflected in future comparisons without leaving stale vectors.

Concepts

From Prose Document to Searchable Vector Index

An approved scope document is a human-readable artifact — sections of prose describing deliverables, exclusions, and assumptions. To compare an arbitrary freeform request against it semantically, the document must first be transformed into a form the database can reason about geometrically. The key insight is that you do not embed the whole document as one vector; you chunk it at section boundaries so each vector captures one coherent topic. When a request arrives, the database does not scan for keyword matches — it finds the baseline section whose vector points in the most similar direction in high-dimensional space. This is what makes the approach robust to paraphrasing: a request for "real-time dashboard updates" and a scope section titled "live reporting interface" share geometric proximity even though they share no words.

Storing vectors in pgvector rather than an in-memory index means the baseline survives restarts and scales alongside the application's existing PostgreSQL connection pool — no additional infrastructure is required.

Cosine Similarity as a Delivery Signal

pgvector's <=> operator returns cosine distance — zero when two vectors are identical, one when they are orthogonal. Subtracting from 1 inverts this into a similarity score, which is easier to reason about in a delivery context: 1 means the request is semantically indistinguishable from approved scope, 0 means it shares no thematic overlap. The THRESHOLDS dict sets two cut-points: 0.85 for in_scope and 0.65 for adjacent (see Code Walkthrough). These are not arbitrary — they reflect a deliberate asymmetry. The in_scope band is tight (≥ 0.85) because a false positive here lets unauthorized work proceed without a conversation. The adjacent band (0.65–0.85) creates a middle channel for related-but-distinct requests that need discussion before commitment, rather than forcing a binary in/out decision that would generate friction on legitimately boundary-straddling work.

The Detector as a Governance Boundary

The classification output is not just a label — it is a governance intervention point. Without automated detection, scope creep compounds incrementally: each informal "sure, we can add that" from an engineer or account manager is small in isolation, but the accumulation surfaces only when budget overruns are already locked in. ScopeChangeDetector is designed to sit upstream of any engineering commitment, intercepting every client request before work begins.

The closest_section and closest_section_text fields returned by check_scope_change serve a specific purpose in this workflow: they provide auditable evidence for scope negotiation. Rather than telling a client "that's out of scope" based on subjective recollection, the system can point to the exact section of the approved baseline it compared against — grounding the conversation in contract language. For out_of_scope results, this evidence becomes the starting point for a formal scope change request and budget impact analysis, which is the hard governance boundary the detector enforces before any engineering work begins.

Code Walkthrough

Now that you understand how vector similarity and cosine distance form the backbone of scope drift detection, let's trace how those ideas connect in running code.

The detector is structured as two cooperating classes. ScopeVectorStore owns the embedding pipeline — it chunks an approved scope document by section headers, calls OpenAI's text-embedding-3-small model once per chunk, and persists each vector in a pgvector-backed PostgreSQL table. ScopeChangeDetector owns the classification logic — it embeds an incoming request, queries pgvector for the nearest scope sections using the <=> cosine distance operator, and maps the closest similarity score to one of three delivery-workflow actions.

Code snippetpython
1import openai
2import asyncpg
3from typing import List
4
5class ScopeVectorStore:
6    """Manages scope baseline embeddings in pgvector."""
7
8    def __init__(self, proxy_url: str, db_pool: asyncpg.Pool):
9        self.client = openai.OpenAI(
10            api_key="student-token",
11            base_url=proxy_url
12        )
13        self.pool = db_pool
14
15    def _embed(self, text: str) -> List[float]:
16        response = self.client.embeddings.create(
17            model="text-embedding-3-small",
18            input=text
19        )
20        return response.data[0].embedding
21
22    async def store_baseline(self, engagement_id: str, scope_document: str):
23        """Embed each section of the approved scope document and persist."""
24        sections = [c for c in scope_document.split("\n## ") if c.strip()]
25        async with self.pool.acquire() as conn:
26            await conn.execute(
27                "DELETE FROM scope_vectors WHERE engagement_id = $1",
28                engagement_id,
29            )
30            for idx, section in enumerate(sections):
31                vec = self._embed(section)
32                await conn.execute(
33                    """
34                    INSERT INTO scope_vectors
35                        (engagement_id, section_id, section_text, embedding)
36                    VALUES ($1, $2, $3, $4::vector)
37                    """,
38                    engagement_id, f"s{idx}", section, str(vec),
39                )

store_baseline deletes any previous baseline before inserting, which supports re-baselining after formally approved scope amendments without leaving stale vectors that would corrupt future comparisons.

Code snippetpython
1THRESHOLDS = {"in_scope": 0.85, "adjacent": 0.65}
2
3class ScopeChangeDetector:
4    """Classifies incoming requests against an embedded scope baseline."""
5
6    def __init__(self, vector_store: ScopeVectorStore):
7        self.store = vector_store
8
9    async def check_scope_change(
10        self, engagement_id: str, request_text: str
11    ) -> dict:
12        """Return classification and closest matching scope section."""
13        embedding = self.store._embed(request_text)
14        async with self.store.pool.acquire() as conn:
15            rows = await conn.fetch(
16                """
17                SELECT section_id, section_text,
18                       1 - (embedding <=> $2::vector) AS similarity
19                FROM scope_vectors
20                WHERE engagement_id = $1
21                ORDER BY similarity DESC
22                LIMIT 3
23                """,
24                engagement_id, str(embedding),
25            )
26        if not rows:
27            return {"classification": "no_baseline", "max_similarity": 0.0}
28
29        best = rows[0]
30        sim = float(best["similarity"])
31        if sim >= THRESHOLDS["in_scope"]:
32            classification = "in_scope"
33        elif sim >= THRESHOLDS["adjacent"]:
34            classification = "adjacent"
35        else:
36            classification = "out_of_scope"
37
38        return {
39            "classification": classification,
40            "max_similarity": round(sim, 4),
41            "closest_section": best["section_id"],
42            "closest_section_text": best["section_text"],
43        }

The <=> operator is pgvector's cosine distance notation; subtracting from 1 converts it to a similarity score between 0 and 1. Requests classified as out_of_scope should trigger a formal scope change request and budget impact analysis before any engineering work begins — this is the governance boundary the detector enforces.

Confirm that calling check_scope_change with a clearly in-scope request returns a classification of "in_scope" with max_similarity above 0.85, and that an unrelated request returns "out_of_scope" with a score below 0.65.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do delete the existing baseline with DELETE FROM scope_vectors WHERE engagement_id = $1 before calling store_baseline again — without this purge, re-baselining after a formally approved scope amendment leaves stale section vectors in pgvector that compete with the updated chunks, producing phantom high-similarity matches that misclassify new requests as in-scope.
2. ✓Do convert pgvector's <=> cosine distance to similarity via 1 - (embedding <=> $2::vector) before comparing against THRESHOLDS — <=> returns a distance where 0 means identical and 1 means orthogonal; comparing the raw distance against THRESHOLDS["in_scope"] (0.85) without inversion reverses the classification logic entirely, marking the most dissimilar requests as in-scope.
3. ✓Do use out_of_scope classifications as a hard governance gate that requires a formal scope change request and budget impact analysis before any engineering work begins — the detector's value is catching scope creep before commitments are made, not after; routing out_of_scope results directly to an engineering queue defeats the delivery-risk purpose of the classification.

Don'ts

1. ✗Don't treat adjacent classifications (similarity 0.65–0.84) as implicitly in-scope — adjacent means the request is semantically related to an indexed section but not covered by it; proceeding without a governance review is precisely the silent-commitment pattern the three-zone model (in_scope / adjacent / out_of_scope) was designed to surface and route differently.
2. ✗Don't skip re-embedding the baseline after changing the \n## section-chunking delimiter in store_baseline — the pgvector nearest-neighbor query in check_scope_change compares an incoming request's embedding against section-level vectors; if the baseline was indexed under a different chunking boundary, the stored vectors represent different text units and the similarity scores become meaningless against the new chunk shape.
3. ✗Don't pass the raw <=> distance value in max_similarity back to the caller without the 1 - inversion — downstream governance workflows that read max_similarity to decide escalation thresholds will interpret a distance of 0.9 (nearly orthogonal) as high confidence of a match, silently approving out-of-scope requests that should have triggered a budget review.