Encrypt embeddings at rest and in transit

Introduction

When you store embedding vectors on GKE without explicit encryption controls, you rely entirely on Google's default key management—meaning you cannot audit key usage, rotate keys on your own schedule, or revoke access during a security incident. Teams that handle sensitive documents, proprietary model outputs, or regulated data need customer-managed encryption keys (CMEK) for at-rest protection and verified TLS for in-transit protection. By the end of this lesson, you will configure a CMEK-backed StorageClass for pgvector workloads and establish SSL-verified connections between application pods and your vector database, giving you auditable, revocable encryption across both layers.

Key Terminology

• Customer-Managed Encryption Key (CMEK) — a symmetric Cloud KMS key you provision and own that wraps data on GKE persistent disks; unlike Google-managed defaults, every encrypt/decrypt operation appears in Cloud Audit Logs and the key can be revoked independently of the cluster without touching the cluster itself.
• StorageClass — a Kubernetes resource that, when configured with the disk-encryption-kms-key parameter, instructs the GKE pd.csi.storage.gke.io CSI driver to wrap every PersistentVolume's disk with your CMEK; create_cmek_storage_class registers one named pgvector-cmek using kubernetes.client.StorageV1Api.
• Cloud KMS Key Ring — an organizational container in Cloud KMS that groups related crypto keys by project and region; create_cmek_storage_class creates a key ring before attaching a ENCRYPT_DECRYPT-purpose CryptoKey to it, and the key ring's resource name forms the prefix of the full key name passed to the StorageClass.
• sslmode="verify-full" — the psycopg2 connection parameter that enforces both certificate-chain validation against sslrootcert and hostname verification; stronger than "require" (no cert check) or "verify-ca" (no hostname check), it closes the class of attacks where a valid but mismatched certificate passes weaker modes.
• Encryption boundary — one of the two independent protection layers the lesson wires together: CMEK-encrypted persistent disks at rest and verified TLS in transit; each boundary is cryptographically independent, so compromising one does not automatically expose data protected by the other.
• pg_stat_ssl — the PostgreSQL system view used to confirm a live connection is actually using TLS; querying SELECT ssl, version FROM pg_stat_ssl WHERE pid = pg_backend_pid() and seeing ssl = t and version = TLSv1.3 is the verification step the lesson specifies after wiring connect_pgvector_tls.

Concepts

Why Two Independent Encryption Layers

Encryption for pgvector workloads on GKE must address two distinct threat scenarios that require entirely different controls. At-rest encryption protects against physical or administrative access to the underlying storage—a compromised node, a disk snapshot exfiltrated through a cloud operator's access, or a misconfigured IAM policy that exposes a GKE node's filesystem. In-transit encryption protects against network-layer interception—an attacker on the pod overlay network, a misconfigured Kubernetes NetworkPolicy, or a man-in-the-middle inside the cluster.

These threats are addressed by mechanisms that share no cryptographic material and no control plane: a Cloud KMS CMEK wraps disk data at rest, and TLS with certificate verification secures the wire between your application pod and pgvector. This independence is intentional. Compromising one layer—say, intercepting an unencrypted connection—does not touch the CMEK wrapping the disk, and revoking the KMS key does not retroactively expose already-intercepted traffic. Each layer can also be audited, rotated, and revoked on its own schedule without disrupting the other, which is precisely what compliance frameworks requiring customer-controlled key management are looking for.

How CMEK Flows Through Kubernetes

The path from a Cloud KMS key to an encrypted persistent disk passes through four ordered steps (see Code Walkthrough):

kms_v1.KeyManagementServiceClient first provisions a key ring and a ENCRYPT_DECRYPT-purpose CryptoKey in the target region. The resulting resource name—e.g. projects/my-proj/locations/us-central1/keyRings/my-ring/cryptoKeys/my-key—is the handle used in every subsequent step. That name is embedded in a Kubernetes StorageClass as the disk-encryption-kms-key parameter. Any PersistentVolumeClaim that references this StorageClass causes the GKE CSI driver to call Cloud KMS and wrap the disk's per-disk data encryption key (DEK) before writing it to the PD metadata. Cloud KMS holds only the key encryption key (KEK); the disk stores data encrypted with its DEK. Revoking the CMEK therefore renders the DEK unrecoverable without touching the disk's raw bytes—which is exactly what "revocable encryption" means in practice.

Why verify-full Is the Only Safe Mode Inside a Cluster

psycopg2 exposes four sslmode levels, and each embodies a different threat model. disable sends plaintext. require encrypts the wire but accepts any certificate—including one self-signed by an attacker—so a man-in-the-middle inside the cluster can still intercept the session. verify-ca checks that the server certificate was signed by the CA at sslrootcert, but it does not verify the hostname. Inside a GKE cluster where multiple services may share a single internal CA, verify-ca alone is insufficient: any pod holding a certificate from that CA could impersonate the pgvector service. verify-full closes this gap by validating the full certificate chain and matching the server's hostname against the certificate's CN or SAN fields, ensuring the connection terminates at exactly the host named in the call to connect_pgvector_tls.

The sslrootcert parameter points to the cluster's internal CA bundle, which is mounted into the application pod from a Kubernetes Secret volume rather than baked into the container image. This separation means the CA can be rotated without rebuilding the image—only the Secret and the pod need to be updated—keeping certificate lifecycle management decoupled from the deployment pipeline.

Code Walkthrough

Now that you understand the two independent encryption layers—CMEK-encrypted persistent disks at rest and verified TLS in transit—the following code demonstrates how to wire them together programmatically.

The first step is provisioning a Cloud KMS key and a CMEK-backed Kubernetes StorageClass that your pgvector PersistentVolumeClaim will reference. The create_cmek_storage_class function calls kms_v1.KeyManagementServiceClient to create a key ring and a symmetric encryption key in your chosen region, then uses kubernetes.client.StorageV1Api to register a StorageClass that passes the KMS key name as the disk-encryption-kms-key parameter to the GKE CSI driver. Any PVC that uses this StorageClass will have its underlying persistent disk wrapped with your customer-managed key, making key usage visible in Cloud Audit Logs and revocable independently of the cluster.

Code snippetpython
1from google.cloud import kms_v1
2from kubernetes import client, config
3
4def create_cmek_storage_class(
5    project_id: str, location: str, key_ring_id: str, key_id: str
6) -> str:
7    kms_client = kms_v1.KeyManagementServiceClient()
8    parent = f"projects/{project_id}/locations/{location}"
9
10    key_ring = kms_client.create_key_ring(
11        request={"parent": parent, "key_ring_id": key_ring_id, "key_ring": {}}
12    )
13    kms_client.create_crypto_key(
14        request={
15            "parent": key_ring.name,
16            "crypto_key_id": key_id,
17            "crypto_key": {
18                "purpose": kms_v1.CryptoKey.CryptoKeyPurpose.ENCRYPT_DECRYPT
19            },
20        }
21    )
22    key_name = f"{key_ring.name}/cryptoKeys/{key_id}"
23
24    config.load_kube_config()
25    storage_v1 = client.StorageV1Api()
26    storage_class = client.V1StorageClass(
27        metadata=client.V1ObjectMeta(name="pgvector-cmek"),
28        provisioner="pd.csi.storage.gke.io",
29        parameters={
30            "type": "pd-ssd",
31            "disk-encryption-kms-key": key_name,
32        },
33        reclaim_policy="Retain",
34    )
35    storage_v1.create_storage_class(body=storage_class)
36    return key_name

With the CMEK StorageClass registered, the second enforcement point is the connection from your application pod to pgvector. Using sslmode="verify-full" tells the psycopg2 driver to validate the server certificate against a trusted CA, preventing man-in-the-middle attacks inside the cluster. The sslrootcert parameter points to the cluster's internal CA bundle, which you mount into the application pod as a Kubernetes Secret volume.

Code snippetpython
1import psycopg2
2
3def connect_pgvector_tls(
4    host: str, dbname: str, user: str, password: str, ssl_cert_path: str
5) -> psycopg2.extensions.connection:
6    return psycopg2.connect(
7        host=host,
8        dbname=dbname,
9        user=user,
10        password=password,
11        sslmode="verify-full",
12        sslrootcert=ssl_cert_path,
13    )

Choosing sslmode="verify-full" rather than "require" or "verify-ca" enforces hostname verification in addition to certificate chain validation, closing the class of attacks where a valid but mismatched certificate passes weaker checks. The CMEK key and the SSL connection operate independently—compromising one layer does not automatically compromise the other—so both must be active for end-to-end protection.

You'll know it works when gcloud kms keys describe <KEY_NAME> shows state: ENABLED, and a psycopg2 connection running SELECT ssl, version FROM pg_stat_ssl WHERE pid = pg_backend_pid() returns ssl = t and version = TLSv1.3.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do pass the KMS key name to the disk-encryption-kms-key parameter in your pgvector-cmek StorageClass — this is the field the GKE CSI driver reads to wrap each new persistent disk with your customer-managed key, making every write auditable in Cloud Audit Logs and revocable independently of the cluster; omitting it silently falls back to Google-managed keys you cannot rotate or revoke on your own schedule.
2. ✓Do set sslmode="verify-full" in every psycopg2.connect call to pgvector — unlike "require" (no cert validation) or "verify-ca" (no hostname check), "verify-full" enforces both certificate chain validation and hostname verification together, which is the only mode that closes the class of attacks where a valid but mismatched certificate passes weaker checks inside the cluster.
3. ✓Do mount the cluster's internal CA bundle as a Kubernetes Secret volume and supply its path to sslrootcert — this gives verify-full an authoritative trust anchor without baking the certificate material into the container image, ensuring the trust chain stays revocable and separately managed from application code.

Don'ts

1. ✗Don't rely on CMEK at rest as a substitute for verified TLS in transit — the lesson is explicit that the KMS-backed StorageClass and sslmode="verify-full" operate as independent layers; CMEK-encrypted disk blocks do nothing to protect embedding vectors moving over the network between your application pod and pgvector, so both layers must be active simultaneously.
2. ✗Don't substitute sslmode="require" or "verify-ca" for "verify-full" in psycopg2.connect — "require" skips certificate validation entirely and "verify-ca" stops short of hostname verification, meaning a valid certificate from a different host in the cluster still passes, leaving a man-in-the-middle attack surface that "verify-full" explicitly closes.
3. ✗Don't create a StorageClass without the disk-encryption-kms-key parameter and assume it will use your KMS key — the GKE CSI driver only applies CMEK wrapping when that parameter is present; a StorageClass using provisioner: pd.csi.storage.gke.io without it provisions Google-managed keys that produce no entries in Cloud Audit Logs and cannot be independently revoked if a key is compromised.