Deploy RAG defense system on GKE with pgvector

Introduction

In production, a RAG system is only as trustworthy as the controls that protect its knowledge base at runtime — but document fingerprinting, embedding-drift monitoring, canary documents, and provenance tracking deliver no value if they run as ad-hoc scripts on a developer machine instead of a hardened, monitored cluster tier. This lesson teaches you how to deploy the full RAG defense stack on Google Kubernetes Engine: a dedicated rag-defense namespace with Pod Security Admission enforcement, a pgvector StatefulSet backed by a customer-managed-key disk, defense-service Deployments wired to Workload Identity, and the operating discipline — pinned image digests and CronJob alerting — that keeps the defense plane reliable around the clock.

Key Terminology

• Pod Security Admission (PSA) — Kubernetes' built-in admission controller, activated by the pod-security.kubernetes.io/enforce: restricted label on the rag-defense namespace; at restricted level it rejects pods that run as root, mount host paths, or omit mandatory security contexts.
• Audit label — The pod-security.kubernetes.io/audit: restricted namespace label, which records policy violations to the Kubernetes audit log regardless of enforcement state, preserving a forensic trail when enforce is temporarily relaxed during incident response.
• Image digest pinning — Referencing a container image by its immutable @sha256:<hash> suffix (e.g., pgvector/pgvector@sha256:<pin-digest-here>) instead of a mutable tag, guaranteeing that a registry re-push cannot silently replace what the cluster runs.
• Customer-Managed Encryption Key (CMEK) disk — A GKE persistent disk whose encryption key is controlled by the operator rather than Google, provisioned via storageClassName: pd-ssd-cmek so the pgvector data volume cannot be decrypted without the operator's key material.
• Workload Identity — A GKE binding between a Kubernetes ServiceAccount (here provenance-tracker-sa) and a Google Cloud IAM service account, granting pods short-lived, automatically rotated GCP credentials without embedding long-lived keys as Secrets.
• Defense plane — The runtime cluster tier — pgvector StatefulSet, provenance-tracker Deployment, and supporting CronJobs — that enforces RAG poisoning defenses; verified healthy by check_defense_plane() before live ingest traffic is routed through it.

Concepts

Namespace Isolation as a Security Boundary

Kubernetes namespaces are administrative partitions, not security boundaries by default — any pod can reach any other pod across namespaces unless network policies and admission controls say otherwise. The rag-defense namespace hardens that boundary in two complementary ways: the pod-security.kubernetes.io/enforce: restricted label makes the API server itself reject non-compliant pod specs before they are scheduled, and the pod-security.kubernetes.io/audit: restricted label writes policy violations to the audit log independently of enforcement state. The audit label is the more operationally important of the two: under incident pressure, teams often loosen enforce to unblock a rollout, but without audit still set to restricted that window leaves no trace of what the cluster would have caught. Running the defense workloads — pgvector, provenance-tracker, and alerting CronJobs — inside this dedicated namespace means a misconfigured network policy or a compromised application pod in a neighboring namespace cannot silently reach the defense store (see Code Walkthrough).

Image Digest Pinning and Supply-Chain Integrity

Container tags like :latest or :1.4 are mutable pointers: a registry operator can push a new image layer under the same tag at any time, and the next rollout will silently pull the replacement. For a defense plane this is unacceptable — an attacker who can push to the upstream pgvector/pgvector repository could install a tampered image across every cluster that pulls by tag. Image digests (@sha256:<hash>) are content-addressed and immutable: the digest is a cryptographic hash of the image manifest, so the only way to change what the cluster runs is to update the manifest through a deliberate, reviewable commit. This is why both the pgvector StatefulSet and the provenance-tracker Deployment use @sha256:<pin-digest-here> references rather than tags; digest updates should go through the same change-management path as a database schema migration.

Workload Identity and Least-Privilege Access

The provenance-tracker Deployment needs access to GCP services — Cloud Storage for provenance logs, Cloud KMS for CMEK operations — but should never hold a long-lived JSON key. Workload Identity solves this by binding the Kubernetes ServiceAccount named provenance-tracker-sa to a GCP IAM service account, so pods automatically receive short-lived, rotated GCP credentials scoped to only the IAM roles explicitly granted to that account. No credential is stored in a Kubernetes Secret, and revoking access is a single IAM binding change rather than a secret-rotation cycle. Combined with namespace isolation and PSA enforcement, this creates a least-privilege posture where even a compromised defense pod cannot escalate to broader GCP resources.

Operating Discipline: Audit Trail, CronJob Alerting, and Health Verification

Deploying the defense plane is the start, not the end. Three operating practices keep it reliable: first, maintaining the pod-security.kubernetes.io/audit label even when enforce is relaxed, so any policy violation during an incident window appears in the audit log; second, wrapping CronJob-based monitors — drift detection, canary checks — with Alertmanager rules that page on missed successful runs, because a CronJob that silently stops firing provides no defense; and third, running check_defense_plane() before routing live ingest traffic to confirm every pod is in Running phase with all containerStatuses[].ready == True. The liveness and readiness probes on the provenance-tracker container feed this last check: liveness restarts a deadlocked pod, readiness gates traffic, and together they ensure the health signal the script reads is meaningful (see Code Walkthrough).

Code Walkthrough

With the operating-discipline concepts in hand, the manifest below consolidates the three foundational deployment objects — a hardened namespace, a pgvector StatefulSet pinned by image digest to a customer-managed-key disk, and a provenance-tracker Deployment wired to Workload Identity — into a single file you apply with kubectl apply -f rag-defense-stack.yaml:

Code snippetyaml
1---
2# Hardened namespace — Pod Security Admission at restricted level
3apiVersion: v1
4kind: Namespace
5metadata:
6  name: rag-defense
7  labels:
8    pod-security.kubernetes.io/enforce: restricted
9    pod-security.kubernetes.io/audit: restricted   # preserved even if enforce is loosened
10    workload: ai-security
11---
12# pgvector StatefulSet — pinned digest, CMEK disk, dedicated node pool
13apiVersion: apps/v1
14kind: StatefulSet
15metadata:
16  name: pgvector
17  namespace: rag-defense
18spec:
19  serviceName: pgvector
20  replicas: 1
21  selector:
22    matchLabels: {app: pgvector}
23  template:
24    metadata:
25      labels: {app: pgvector}
26    spec:
27      nodeSelector: {workload: ai-security-db}
28      containers:
29      - name: pgvector
30        image: pgvector/pgvector@sha256:<pin-digest-here>   # never a floating tag
31        env:
32        - name: POSTGRES_PASSWORD
33          valueFrom:
34            secretKeyRef: {name: pgvector-secret, key: password}
35        volumeMounts:
36        - {name: data, mountPath: /var/lib/postgresql/data}
37        resources:
38          requests: {cpu: "1", memory: 4Gi}
39          limits:   {cpu: "2", memory: 8Gi}
40  volumeClaimTemplates:
41  - metadata: {name: data}
42    spec:
43      accessModes: [ReadWriteOnce]
44      storageClassName: pd-ssd-cmek
45      resources: {requests: {storage: 100Gi}}
46---
47# Provenance-tracker Deployment — Workload Identity, liveness + readiness probes
48apiVersion: apps/v1
49kind: Deployment
50metadata:
51  name: provenance-tracker
52  namespace: rag-defense
53spec:
54  replicas: 2
55  selector:
56    matchLabels: {app: provenance-tracker}
57  template:
58    metadata:
59      labels: {app: provenance-tracker}
60    spec:
61      serviceAccountName: provenance-tracker-sa
62      containers:
63      - name: app
64        image: registry.example.com/rag-defense/provenance@sha256:<pin-digest-here>
65        env:
66        - {name: PGVECTOR_HOST, value: pgvector.rag-defense.svc.cluster.local}
67        - {name: OTEL_EXPORTER_OTLP_ENDPOINT, value: http://otel-collector:4317}
68        livenessProbe:
69          httpGet: {path: /healthz, port: 8080}
70          initialDelaySeconds: 10
71        readinessProbe:
72          httpGet: {path: /readyz, port: 8080}
73          initialDelaySeconds: 3
74        resources:
75          requests: {cpu: 200m, memory: 256Mi}
76          limits:   {cpu: 500m, memory: 512Mi}

The pod-security.kubernetes.io/audit: restricted label is easy to skip under incident pressure, but it is what gives you the audit trail when the enforce label is temporarily relaxed — exactly the pitfall the operating-discipline section flags. Every image reference uses a digest rather than a tag so a :latest re-push on the upstream registry cannot silently change what your cluster runs.

After applying the manifest, verify the defense plane is healthy before routing live ingest traffic through it:

Code snippetpython
1import subprocess, json, sys
2
3def check_defense_plane(namespace: str = "rag-defense") -> None:
4    """Exit 1 if any pod in the defense namespace is not Running and fully Ready."""
5    result = subprocess.run(
6        ["kubectl", "get", "pods", "-n", namespace, "-o", "json"],
7        capture_output=True, text=True, check=True,
8    )
9    pods = json.loads(result.stdout)["items"]
10    not_ready = [
11        p["metadata"]["name"]
12        for p in pods
13        if p["status"].get("phase") != "Running"
14        or not all(
15            c["ready"]
16            for c in p["status"].get("containerStatuses", [])
17        )
18    ]
19    if not_ready:
20        print(f"NOT READY: {not_ready}", file=sys.stderr)
21        sys.exit(1)
22    print(f"All {len(pods)} pods in '{namespace}' are Running and Ready.")
23
24if __name__ == "__main__":
25    check_defense_plane()

You've completed this when kubectl get pods -n rag-defense shows every pod in Running state with all containers reporting Ready: True, and the script above exits with code 0.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do retain the pod-security.kubernetes.io/audit: restricted label even when temporarily relaxing the enforce label — without the audit label, violations during an incident window are silently swallowed, leaving you no trail to reconstruct what ran in rag-defense while enforcement was loosened.
2. ✓Do pin every image reference in rag-defense-stack.yaml to a SHA256 digest instead of a tag — a floating tag like :latest means a upstream re-push silently replaces what your pgvector StatefulSet or provenance-tracker Deployment runs, bypassing the entire supply-chain control the namespace hardening is designed to enforce.
3. ✓Do run check_defense_plane() (or kubectl get pods -n rag-defense) before routing live ingest traffic through the defense stack — fingerprinting, drift monitoring, canary documents, and provenance tracking all fail silently if the pods that back them are crash-looping or not yet Ready, turning your hardened namespace into a false-confidence layer.

Don'ts

1. ✗Don't schedule the rag-defense namespace's workloads onto general-purpose node pools — the pgvector StatefulSet's nodeSelector: {workload: ai-security-db} exists to isolate CMEK-disk-backed database pods from shared workloads; omitting it allows other tenant pods to share the node and potentially read memory or storage belonging to the defense plane.
2. ✗Don't mount pgvector's persistent volume without the pd-ssd-cmek StorageClass — using a default StorageClass creates a disk outside the customer-managed-key envelope, meaning the vector store holding document fingerprints and provenance records is not covered by your key-rotation and revocation controls.
3. ✗Don't wire the provenance-tracker Deployment to a long-lived service-account key file instead of serviceAccountName: provenance-tracker-sa — bypassing Workload Identity means rotating or revoking GCP credentials requires a manual secret update and pod restart, and a leaked key file grants persistent access to the provenance store that no cluster-level control can revoke without redeployment.