Deploy secrets infrastructure on GKE with Workload Identity

Introduction

When you store LLM API keys as environment variables or mount service-account JSON files into pods, a single log line or snapshot exposes credentials that persist long after the pod is gone. This lesson covers the Kubernetes-side configuration that eliminates long-lived credentials: annotating a ServiceAccount for Workload Identity, wiring External Secrets Operator to sync keys from Google Secret Manager into a namespace-scoped secret, and locking that secret down with a tight RBAC binding. By the end, you'll be able to deploy the complete secrets infrastructure on GKE so provider keys reach gateway pods automatically, refresh within 30 minutes of a GSM rotation, and remain readable by exactly one ServiceAccount.

Key Terminology

• Workload Identity — GKE's mechanism for issuing short-lived GCP OAuth tokens to pods through the instance metadata server, eliminating the need for mounted JSON key files by linking a Kubernetes ServiceAccount (via the iam.gke.io/gcp-service-account annotation) to a Google service account (GSA) secured with an IAM binding.
• ExternalSecret — An ESO custom resource that defines a secret synchronization job: which ClusterSecretStore to read from, which upstream GSM keys to fetch (e.g. providers/openai-api-key), what name to give the resulting K8s Secret, and how frequently to reconcile via refreshInterval.
• ClusterSecretStore — A cluster-scoped ESO resource that holds connection configuration for an external secret backend (here, Google Secret Manager); referenced by name in ExternalSecret.spec.secretStoreRef and shared across namespaces.
• refreshInterval — The ESO polling cadence controlling how quickly a rotation in Google Secret Manager propagates to the in-cluster K8s Secret; the lesson sets this to 30m so gateway pods receive updated provider keys within 30 minutes without a pod restart.
• creationPolicy: Owner — An ExternalSecret target policy that makes ESO the sole owner of the synced K8s Secret, reverting any out-of-band kubectl edits on the next reconciliation cycle and preventing configuration drift.
• resourceNames constraint — A Kubernetes RBAC Role rule field that restricts secret access to a single named secret (e.g., litellm-provider-keys) rather than every secret in the namespace; its omission is the most common misconfiguration when pairing ESO sync with RBAC lockdown.

Concepts

The Two-Sided Workload Identity Handshake

Workload Identity works through a handshake that requires two independent, correctly configured pieces — and failure of either half produces a different, non-obvious error. The Kubernetes side is the iam.gke.io/gcp-service-account annotation on the ServiceAccount; this tells the GKE metadata server which GSA the pod intends to impersonate. The GCP side is an IAM binding that grants the K8s service account the roles/iam.workloadIdentityUser role on that GSA, using the member string serviceAccount:PROJECT.svc.id.goog[namespace/sa-name].

When both are present, pods do not need credentials on disk at all — the metadata server exchanges the pod's K8s identity for a short-lived OAuth token transparently, just like it does for GCE VMs. When only the annotation is present, the metadata server recognizes the request but GSM returns a 403 because the IAM binding is missing. When only the IAM binding is present, the metadata server never issues a token because it does not know which GSA the pod is claiming. Treating both sides as a matched pair — and confirming the member-string namespace and SA name are exact — is the primary operational discipline this lesson builds (see Code Walkthrough).

ESO's Pull-and-Own Sync Model

External Secrets Operator does not passively watch GSM for push notifications. It polls on a refreshInterval cadence, reads the specified secret versions from GSM, and writes the result into a K8s Secret in the target namespace. This pull model means rotation latency is bounded by refreshInterval (30 minutes here), and gateway pods pick up new keys without a rolling restart.

creationPolicy: Owner gives ESO authoritative ownership of the synced Secret. Any manual edit — a direct kubectl patch secret litellm-provider-keys — is silently reverted on the next reconciliation. This is not a footgun to avoid; it is the intended property. Configuration drift in secrets infrastructure is a class of incident on its own, and Owner policy is what closes that surface. The ClusterSecretStore is the named bridge that tells ESO which backend to authenticate against; ESO uses the secrets-reader GSA credentials (obtained via Workload Identity) to talk to GSM, completing the trust chain from pod identity through to the upstream key store.

RBAC as the Namespace Boundary

The K8s Secret that ESO writes is only as protected as the RBAC rules guarding it. A Role without a resourceNames field grants get on every Secret in the namespace — a blast radius that silently grows as new Secrets are added. Naming litellm-provider-keys explicitly in resourceNames makes the permission declarative, auditable, and minimal: the gateway ServiceAccount can read exactly one Secret.

The verification pattern confirms both the positive and negative cases: kubectl auth can-i get secret/litellm-provider-keys returning yes proves the binding works; kubectl auth can-i list secrets returning no proves the scope is tight. Both checks must pass — a yes on the second means the namespace is over-permissioned (see Code Walkthrough). Running only the positive check is a common gap that leaves broad list access undetected in production clusters.

Code Walkthrough

Now that you understand the Workload Identity binding pattern, the ESO sync model, and the RBAC pitfalls the Concepts section calls out, here is the minimal Kubernetes configuration that wires them together.

The foundation is a ServiceAccount annotated with the Google service account (GSA) identity. This annotation is the only K8s-side signal the GKE metadata server needs to issue short-lived tokens on the pod's behalf — no JSON key file mounted, no long-lived credential on disk.

Code snippetyaml
1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  name: secrets-reader
5  namespace: ai-llm-gateway
6  annotations:
7    iam.gke.io/gcp-service-account: secrets-reader@PROJECT.iam.gserviceaccount.com

On the GCP side you apply one gcloud iam service-accounts add-iam-policy-binding call granting the K8s SA the roles/iam.workloadIdentityUser role on the GSA, with the member string serviceAccount:PROJECT.svc.id.goog[ai-llm-gateway/secrets-reader]. Both sides must be present: the annotation without the IAM binding fails with 403 at token-exchange time; the binding without the annotation means the metadata server does not know which GSA the pod intends to impersonate.

With identity established, External Secrets Operator syncs the secret from Google Secret Manager into the workload namespace. A Role with resourceNames locks down who can read the result:

Code snippetyaml
1apiVersion: external-secrets.io/v1beta1
2kind: ExternalSecret
3metadata:
4  name: provider-keys
5  namespace: ai-llm-gateway
6spec:
7  refreshInterval: 30m
8  secretStoreRef:
9    kind: ClusterSecretStore
10    name: gsm-cluster-store
11  target:
12    name: litellm-provider-keys
13    creationPolicy: Owner
14  data:
15  - secretKey: openai_api_key
16    remoteRef:
17      key: providers/openai-api-key
18      version: latest
19---
20apiVersion: rbac.authorization.k8s.io/v1
21kind: Role
22metadata:
23  namespace: ai-llm-gateway
24  name: provider-keys-reader
25rules:
26- apiGroups: [""]
27  resources: ["secrets"]
28  resourceNames: ["litellm-provider-keys"]
29  verbs: ["get"]

refreshInterval: 30m keeps the synced secret current with upstream rotations — a key changed in GSM reaches gateway pods within 30 minutes without a pod restart. creationPolicy: Owner means ESO owns the lifecycle of litellm-provider-keys; any manual edit to the K8s Secret is reverted on the next reconciliation, eliminating configuration drift. The resourceNames constraint in the Role is the detail the Concepts section flags as the most common omission: without it, the binding grants read access to every secret in the namespace rather than the one intended.

Confirm that kubectl auth can-i get secret/litellm-provider-keys --namespace ai-llm-gateway --as system:serviceaccount:ai-llm-gateway:litellm-sa returns yes and that the same command with list secrets returns no — together those two responses verify that Workload Identity, ESO sync, and RBAC lockdown are all functioning as intended.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do annotate the ServiceAccount with iam.gke.io/gcp-service-account AND apply the roles/iam.workloadIdentityUser IAM binding on the GSA — both halves must be present simultaneously; the annotation alone fails with a 403 at token-exchange time because the GKE metadata server finds no IAM binding to authorize the impersonation.
2. ✓Do set creationPolicy: Owner on the ExternalSecret — this makes ESO the authoritative owner of litellm-provider-keys, so any manual edit to the synced K8s Secret is reverted on the next reconciliation cycle and configuration drift is structurally impossible.
3. ✓Do verify the RBAC lockdown with two kubectl auth can-i checks — confirm get secret/litellm-provider-keys returns yes and list secrets returns no for system:serviceaccount:ai-llm-gateway:litellm-sa; the list check is the one that proves resourceNames: ["litellm-provider-keys"] is actually constraining scope rather than granting namespace-wide secret access.

Don'ts

1. ✗Don't omit resourceNames: ["litellm-provider-keys"] from the Role rules — without it the binding grants get on every secret in the ai-llm-gateway namespace, meaning any pod bound to that role can read any credential that ESO or another operator writes there, defeating the single-secret lockdown the RBAC is meant to enforce.
2. ✗Don't mount a service-account JSON key file into pods as a substitute for the iam.gke.io/gcp-service-account annotation — JSON key files are long-lived credentials that persist in logs, snapshots, and kubectl describe output long after the pod is gone, which is the exact exposure model this Workload Identity pattern is designed to eliminate.
3. ✗Don't set refreshInterval longer than 30m on the ExternalSecret if upstream GSM rotation is in use — a stale interval means gateway pods continue presenting a rotated-away key until the next ESO reconciliation, causing live provider 401 errors for the full duration of the lag without any pod restart required to trigger the fix.