Monitor vector store access patterns and anomalies

Introduction

In production, even a fully patched vector store can be drained by a compromised service account that has legitimate access—no injection or encryption failure required, just patient, systematic similarity queries. Preventive controls alone cannot catch this because the attacker never violates a permission boundary; they simply query more than they should, or in patterns that betray enumeration. This lesson teaches you to recognize the behavioral fingerprints of vector store attacks—high-frequency probing, dimensional sweeping, result-set inflation, and off-hours access spikes—and to build a statistical anomaly detection pipeline that issues graduated responses before significant exfiltration occurs.

Key Terminology

• Centroid distance: The Euclidean distance between an incoming query vector and the mean of all stored embeddings for a tenant, used to detect queries probing unusual regions of the embedding space
• Z-score: A statistical measure expressing how many standard deviations a value lies from the historical mean, forming the basis of the anomaly scoring system
• Sliding window: A fixed-duration time bucket (typically 5 minutes) over which raw audit events are aggregated into statistical summaries for baseline comparison
• Dimensional sweeping: An attack technique where an adversary systematically queries across the embedding space in a grid pattern to reconstruct stored embeddings through nearest-neighbor responses
• Graduated response: A defense strategy that applies increasingly restrictive countermeasures (logging → throttling → network isolation) proportional to the anomaly severity score
• Cold-start period: The initial data collection phase (minimum 24 windows) during which the anomaly scorer reports "insufficient_data" instead of scores, preventing false positives for new tenants

Concepts

Threat Model for Vector Store Access Patterns

Before writing detection logic, you must enumerate what abnormal behavior looks like in a vector store context. Unlike traditional databases where anomaly detection focuses on query volume and data volume, vector stores introduce geometric threats unique to embedding spaces.

• High-frequency probing: A single tenant or service account issues far more similarity searches than its historical baseline, suggesting automated enumeration of the embedding space
• Dimensional sweeping: Queries arrive with embedding vectors that systematically cover regions of the vector space in a grid-like pattern, indicating an attempt to reconstruct stored embeddings through nearest-neighbor triangulation
• Cross-tenant leakage attempts: Queries target embedding IDs or metadata filters that belong to other tenants, testing whether row-level access control has gaps
• Result set inflation: A user who normally retrieves top-5 results suddenly requests top-500, or repeatedly adjusts the similarity threshold to pull increasingly distant neighbors
• Off-hours access spikes: Legitimate users follow predictable temporal patterns; compromised credentials often operate during off-hours when security teams have reduced coverage
• Query vector clustering: Normal user queries distribute across the embedding space reflecting diverse information needs; attack queries cluster tightly around specific regions of interest

Graduated Response Strategy

Detection without response is merely logging. The anomaly scoring engine's severity levels map directly to automated responses that progressively restrict a tenant's access.

• Warning (score ≥ 2.0): The system emits a structured log entry to Grafana and sends a Slack notification to the security channel. No query restrictions are applied. The operator has full context—tenant ID, the specific z-score dimensions that fired, and the raw window statistics—to decide whether this is a legitimate usage spike.

• Critical (score ≥ 3.5): The TenantThrottler component injects a rate limit into the QueryAuditMiddleware, restricting the flagged tenant to 10 queries per minute (down from the default 100). The throttle propagates to all GKE pods serving that tenant through the shared Redis state. Simultaneously, the system snapshots the current window's raw events to a GCS bucket for forensic analysis.

• Emergency (score ≥ 5.0): The system invokes the Kubernetes API to patch the tenant's network policy, blocking all egress from their designated pods to the pgvector service. This is the nuclear option—it prevents data exfiltration at the network level while the security team investigates. The corresponding row-level access control tokens for the tenant are revoked, and the embedding integrity registry flags all embeddings accessed during the anomalous window for re-validation.

Connecting Anomaly Detection to Other Security Layers

Anomaly detection does not operate in isolation. Each security mechanism covered in this chapter feeds data into the monitoring pipeline and receives signals from it.

• Row-level access control: When the anomaly scorer flags a tenant at critical severity, the access control layer tightens its token expiry from the default 60 minutes down to 5 minutes. This forces more frequent re-authentication, limiting the blast radius of a compromised token. The audit middleware records which row-level policies were evaluated for each query, enabling the scorer to detect attempts to enumerate policy boundaries.

• Query injection prevention: The injection prevention layer logs every query that triggers its sanitization rules. These sanitization events feed into the anomaly scorer as a separate dimension—a tenant that repeatedly sends queries requiring sanitization is almost certainly probing for injection vulnerabilities. The scorer assigns a sanitization event ratio above 15% as automatically critical regardless of other dimensions.

• Embedding integrity: When the anomaly detection system escalates to emergency, it triggers a targeted integrity check on all embeddings that were returned as results during the anomalous window. If an attacker managed to poison embeddings before detection, this cross-reference catches the contamination during incident response rather than leaving it for the next scheduled integrity sweep.

• Vector store encryption: The encryption layer's key access logs feed into the same Redis stream as query audit events. An anomaly in decryption request patterns—such as a service account requesting decryption keys for tenants it has never served—triggers an independent alert path that bypasses the per-tenant statistical model entirely.

Deploying on GKE with Network Isolation

On GKE, the monitoring components deploy as a sidecar container alongside the application pod, sharing the pod's network namespace but running as a separate process. The Redis stream used for event buffering runs as a dedicated StatefulSet with a network policy restricting ingress to only the application pods and the anomaly scorer deployment. This ensures that even if an attacker compromises a different workload in the cluster, they cannot tamper with audit events or manipulate anomaly baselines.

The TenantThrottler's ability to patch network policies at emergency severity requires a dedicated Kubernetes service account with a narrowly scoped RBAC role—only patch on NetworkPolicy resources in the vector store namespace. This principle of least privilege ensures that a compromised monitoring component cannot escalate its own access beyond its intended response actions.

Code Walkthrough

Now that you have mapped the threat model—high-frequency probing, dimensional sweeping, cross-tenant leakage, and result-set inflation—the implementation follows naturally from each threat's behavioral signature.

The monitoring pipeline has three stages: capture every query as a structured audit event, aggregate those events into per-tenant sliding windows, and score each window against a statistical baseline to produce an anomaly signal. The QueryAuditMiddleware below handles the first stage. It wraps the pgvector query path, records the query vector's L2 norm and its centroid distance from the tenant's mean embedding, and emits a structured event to a Redis stream without blocking the caller.

Code snippetpython
1import json
2import numpy as np
3from dataclasses import dataclass, asdict
4from datetime import datetime, timezone
5import redis.asyncio as aioredis
6
7@dataclass
8class QueryAuditEvent:
9    tenant_id: str
10    user_id: str
11    timestamp: str
12    query_norm: float
13    centroid_distance: float
14    top_k: int
15    similarity_threshold: float
16    latency_ms: float
17    result_count: int
18    source_ip: str
19
20class QueryAuditMiddleware:
21    STREAM_KEY = "vector_store:audit_events"
22
23    def __init__(self, redis_url: str, centroid_cache: dict):
24        self._redis = aioredis.from_url(redis_url)
25        self._centroids = centroid_cache  # {tenant_id: np.ndarray}
26
27    def _centroid_distance(self, tenant_id: str, query_vec: np.ndarray) -> float:
28        centroid = self._centroids.get(tenant_id)
29        if centroid is None:
30            return 0.0
31        return float(np.linalg.norm(query_vec - centroid))
32
33    async def record_query(
34        self,
35        tenant_id: str,
36        user_id: str,
37        query_vec: np.ndarray,
38        top_k: int,
39        similarity_threshold: float,
40        result_count: int,
41        latency_ms: float,
42        source_ip: str,
43    ) -> None:
44        event = QueryAuditEvent(
45            tenant_id=tenant_id,
46            user_id=user_id,
47            timestamp=datetime.now(timezone.utc).isoformat(),
48            query_norm=float(np.linalg.norm(query_vec)),
49            centroid_distance=self._centroid_distance(tenant_id, query_vec),
50            top_k=top_k,
51            similarity_threshold=similarity_threshold,
52            latency_ms=latency_ms,
53            result_count=result_count,
54            source_ip=source_ip,
55        )
56        await self._redis.xadd(
57            self.STREAM_KEY, {"data": json.dumps(asdict(event))}
58        )

A separate worker process consumes these events, groups them into 5-minute sliding windows, and computes Z-scores for each behavioral dimension—query frequency, centroid distance, and top_k values—against the tenant's rolling baseline. A cold-start period of at least 24 windows must pass before the scorer emits numeric scores; during this phase it returns "insufficient_data" to prevent false positives for new tenants. Once the baseline is established, scores that cross the warning threshold trigger logging and alerting; scores that cross the critical threshold activate the graduated response chain, advancing through throttling and, if necessary, network isolation of the offending tenant.

The key design constraint is that the monitoring path must never block the query path. Because record_query emits to the Redis stream with a fire-and-forget await, a monitoring pipeline failure leaves vector store queries unaffected—essential for production reliability on GKE.

Check that after instrumenting the middleware and replaying a batch of similarity queries, your Redis stream contains one JSON entry per query with non-null centroid_distance values, and that the anomaly scorer returns "insufficient_data" for any tenant whose window count has not yet reached the 24-window cold-start threshold.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do enforce the 24-window cold-start guard before emitting numeric Z-scores — returning "insufficient_data" until the tenant's rolling baseline has at least 24 sliding windows prevents the anomaly scorer from triggering false-positive alerts on new tenants whose behavioral statistics are too sparse to produce meaningful Z-scores for query frequency, centroid distance, or top_k.
2. ✓Do emit audit events to the Redis stream with a fire-and-forget await in record_query — keeping the monitoring path non-blocking ensures that a Redis failure or a slow consumer worker cannot stall the pgvector query path; the QueryAuditMiddleware must never add latency to the critical query response chain.
3. ✓Do capture centroid_distance from the tenant's mean embedding on every query — this metric is the primary behavioral fingerprint for dimensional sweeping and systematic enumeration; query_norm and query frequency alone cannot distinguish an attacker probing the embedding space from a legitimate burst of similar queries.

Don'ts

1. ✗Don't rely solely on preventive controls such as row-level permissions or encryption to detect exfiltration — a compromised service account that holds legitimate access never trips a permission boundary; only the statistical anomaly scorer, watching for high-frequency probing and result-set inflation patterns across sliding windows, can surface that threat before significant data leaves the store.
2. ✗Don't lower or skip the 24-window cold-start threshold — emitting numeric anomaly scores before the baseline stabilizes causes the graduated response chain to throttle or network-isolate tenants that are simply new, because their Z-scores are dominated by sampling noise rather than genuine behavioral deviation.
3. ✗Don't jump directly to network isolation when the anomaly scorer crosses the critical threshold — the pipeline is designed to advance through throttling first; bypassing graduated escalation removes the correction window needed to distinguish a sustained enumeration campaign from a transient spike that a throttle alone would suppress.