Deploy security infrastructure on GKE

Introduction

Engineers often find that coordinating a multi-service AI platform on Kubernetes becomes a credential-management problem as much as an infrastructure problem: each service needs access to different Cloud APIs, and hardcoded keys create sprawling security debt with no clear audit trail. This lesson shows how to solve both problems at once by deploying the full AI security platform as a Helm umbrella chart while configuring Workload Identity for each service—eliminating key files entirely and enforcing least-privilege access at the IAM layer. By the end, you'll be able to deploy the platform to GKE and verify that each component authenticates to Secret Manager, Cloud Monitoring, and Cloud Storage through short-lived, automatically rotated credentials.

Key Terminology

• Workload Identity — A GKE mechanism that binds a Kubernetes service account to a Google Cloud IAM service account so pods authenticate to Cloud APIs via automatically rotated short-lived OAuth tokens instead of mounted key files.
• IAM service account — A Google Cloud identity (e.g., defense-pipeline-sa@<project>.iam.gserviceaccount.com) created with gcloud iam service-accounts create that represents a workload rather than a human user, and to which IAM roles are attached.
• Workload Identity binding — The IAM policy binding that associates a Kubernetes service account (expressed as serviceAccount:<project>.svc.id.goog[<namespace>/<k8s-sa>]) with a Google Cloud IAM service account using roles/iam.workloadIdentityUser, completing the trust chain between the cluster and Cloud IAM.
• Least-privilege IAM role — A narrowly scoped Cloud IAM role—such as roles/secretmanager.secretAccessor for read-only secret access—assigned individually to each service's IAM account so that a compromised workload can only affect the APIs its role explicitly permits.
• GKE metadata server — The node-level component that intercepts outbound credential requests from pods, resolves the Workload Identity binding for the pod's Kubernetes service account, and returns a short-lived OAuth token scoped to the corresponding IAM service account.
• umbrella chart annotation — The iam.gke.io/gcp-service-account key written into a Kubernetes ServiceAccount manifest via values.yaml, which the GKE metadata server reads to determine which IAM service account to impersonate when issuing tokens to that pod.

Concepts

Why Credential Files Are the Wrong Unit of Trust

When an AI security platform spans multiple services each needing different Cloud APIs, the naive approach is to export service account key files as JSON and mount them as Kubernetes Secrets. This creates credential sprawl: keys live on disk, in etcd, and in backups; they require manual rotation; and if any pod is compromised, the attacker inherits long-lived credentials scoped to everything that key can do. There is also no fine-grained audit trail—Cloud logs record a key ID, not the specific workload that used it.

Workload Identity eliminates the key file as an artifact entirely. Pods never possess a credential on disk; instead, the GKE metadata server intercepts each outbound Cloud API call, resolves which Kubernetes service account the pod is running as, looks up the bound IAM service account, and issues a short-lived OAuth token just-in-time. The token expires in under an hour and is rotated automatically with no operator intervention. Cloud audit logs record the IAM service account name on every request, so calls are attributable to the specific workload—not a shared key ID.

The Three-Step Pattern, Applied Once Per Service

Every service in the umbrella chart follows the same three-step setup that must be completed in Cloud IAM before the Helm chart is applied (see Code Walkthrough):

1. Create the IAM service account (gcloud iam service-accounts create) — establishes the Google Cloud identity that IAM policy will attach to.
2. Grant minimum permissions (gcloud projects add-iam-policy-binding) — each service receives exactly the roles its workload requires. The defense pipeline gets roles/secretmanager.secretAccessor (read-only secret retrieval); the monitoring stack gets roles/monitoring.metricWriter; the compliance API gets roles/storage.objectCreator. No service's IAM account can touch another service's APIs.
3. Add the Workload Identity binding (gcloud iam service-accounts add-iam-policy-binding with roles/iam.workloadIdentityUser) — authorizes the Kubernetes service account running in the ai-security namespace to impersonate the IAM service account.

These three steps configure state in Cloud IAM that persists across redeployments. The Helm chart does not replicate this state; it only references it through a single annotation per service.

Annotations as the Bridge Between Kubernetes and IAM

The Helm umbrella chart's values.yaml centralizes one annotation per service—iam.gke.io/gcp-service-account—pointing at the IAM service account email. When Helm renders the sub-chart templates and applies them to the cluster, that annotation lands on each Kubernetes ServiceAccount object. The GKE metadata server reads it at pod scheduling time to determine which IAM service account to impersonate when the pod requests credentials.

The IAM binding (Cloud-side) and the annotation (cluster-side) must both be present and agree with each other. If the annotation is missing, the metadata server has no IAM account to impersonate. If the binding is missing, the impersonation is rejected even though the annotation is there. Either gap produces a permission-denied error on the pod's first Cloud API call. Running kubectl describe serviceaccount defense-pipeline -n ai-security confirms the annotation was written by Helm; inspecting the IAM policy confirms the binding exists on the Cloud side. Both checks are required before the defense pipeline can successfully read from Secret Manager.

The three-step Workload Identity pattern wires a Kubernetes service account to a Google Cloud identity without key files:

Code Walkthrough

Now that you understand how Workload Identity eliminates credential sprawl by binding Kubernetes service accounts to Google Cloud IAM accounts, you can wire each platform service to exactly the Cloud APIs it needs. The three-step process—create an IAM service account, grant minimum permissions, add the Workload Identity binding—repeats for each service in the umbrella chart.

The following commands configure Workload Identity for the defense pipeline, which requires read-only access to Secret Manager to retrieve LLM API keys:

Code snippetbash
1PROJECT_ID="your-project-id"
2NAMESPACE="ai-security"
3
4# Create the IAM service account for the defense pipeline
5gcloud iam service-accounts create defense-pipeline-sa \
6  --display-name="Defense Pipeline SA" \
7  --project="${PROJECT_ID}"
8
9# Grant read-only Secret Manager access (least privilege)
10gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
11  --member="serviceAccount:defense-pipeline-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
12  --role="roles/secretmanager.secretAccessor"
13
14# Bind the Kubernetes service account to the IAM account
15gcloud iam service-accounts add-iam-policy-binding \
16  "defense-pipeline-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
17  --member="serviceAccount:${PROJECT_ID}.svc.id.goog[${NAMESPACE}/defense-pipeline]" \
18  --role="roles/iam.workloadIdentityUser"

Repeat this pattern for the monitoring stack (using roles/monitoring.metricWriter) and the compliance API (using roles/storage.objectCreator). Once the IAM bindings exist, the Helm umbrella chart propagates the annotation that links each Kubernetes service account to its IAM counterpart. A single values.yaml excerpt defines all three annotations in one place, so sub-chart templates inject them automatically during helm install:

Code snippetyaml
1defensePipeline:
2  serviceAccount:
3    annotations:
4      iam.gke.io/gcp-service-account: "defense-pipeline-sa@your-project-id.iam.gserviceaccount.com"
5
6monitoring:
7  serviceAccount:
8    annotations:
9      iam.gke.io/gcp-service-account: "monitoring-sa@your-project-id.iam.gserviceaccount.com"
10
11complianceApi:
12  serviceAccount:
13    annotations:
14      iam.gke.io/gcp-service-account: "compliance-api-sa@your-project-id.iam.gserviceaccount.com"

When Helm renders the sub-chart templates, these annotations are written into each Kubernetes ServiceAccount manifest. The defense pipeline pod never reads a key file; the GKE metadata server intercepts its credential requests and exchanges the Workload Identity binding for a short-lived OAuth token scoped to Secret Manager reads only. The monitoring stack receives an equally narrow token scoped to metric writes, and the compliance API receives one scoped to Cloud Storage object creation—each service is isolated from the others' APIs at the IAM layer.

Verify by running kubectl describe serviceaccount defense-pipeline -n ai-security and confirming the iam.gke.io/gcp-service-account annotation is present, then hitting the defense pipeline's /health endpoint and checking that its response reports a successful Secret Manager connection rather than an authentication error.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do repeat the three-step Workload Identity pattern—gcloud iam service-accounts create, add-iam-policy-binding for the scoped role, then add-iam-policy-binding for roles/iam.workloadIdentityUser—for every service in the umbrella chart — skipping or reordering any step leaves the Kubernetes service account without a valid token exchange path, causing the GKE metadata server to reject credential requests at runtime.
2. ✓Do assign the narrowest predefined role each service actually calls: roles/secretmanager.secretAccessor for the defense pipeline, roles/monitoring.metricWriter for the monitoring stack, and roles/storage.objectCreator for the compliance API — broader roles like roles/editor silently grant each pod access to every other service's APIs, eliminating the per-service IAM isolation that Workload Identity is designed to enforce.
3. ✓Do centralize all three iam.gke.io/gcp-service-account annotations in the umbrella chart's values.yaml so sub-chart templates inject them automatically during helm install — hardcoding annotations inside individual sub-chart templates breaks the single-source-of-truth model and forces manual edits in multiple files whenever a project ID or service account name changes.

Don'ts

1. ✗Don't hardcode or mount GCP service account key files in pods instead of using Workload Identity — key files create persistent, manually rotated credentials with no automatic audit trail; the entire point of the gcloud iam service-accounts add-iam-policy-binding … --role="roles/iam.workloadIdentityUser" binding is to let the GKE metadata server issue short-lived OAuth tokens so key files never exist on the node.
2. ✗Don't reuse a single IAM service account (e.g., defense-pipeline-sa) for multiple services by granting it the union of all required roles — this collapses the per-service isolation boundary: a compromised defense pipeline pod would inherit roles/monitoring.metricWriter and roles/storage.objectCreator alongside its own secretAccessor, which is exactly the credential-sprawl problem Workload Identity is meant to solve.
3. ✗Don't skip the post-deploy verification step of running kubectl describe serviceaccount defense-pipeline -n ai-security and checking the /health endpoint for a successful Secret Manager connection — a missing iam.gke.io/gcp-service-account annotation (e.g., from a Helm values typo or a forgotten helm upgrade) causes the pod to fall through to the node's default service account and fail silently with authentication errors rather than a clear startup crash.