Implement GKE Workload Identity for secure service auth

Introduction

Engineers often solve the GCP access problem for Kubernetes pods by mounting a JSON service account key as a secret — a key that is long-lived, widely scoped, and completely outside GCP's identity graph once downloaded. A single compromised pod can expose credentials valid for up to ten years and usable from anywhere on the internet. GKE Workload Identity eliminates this attack surface by federating Kubernetes service accounts with GCP IAM, so pods receive short-lived tokens brokered by the GKE metadata server instead of carrying any persistent secret. By the end of this lesson, you'll be able to configure Workload Identity bindings programmatically, verify the resulting authentication chain, and integrate it into the broader GKE security posture.

Key Terminology

• Workload Identity Pool: The identity namespace (PROJECT_ID.svc.id.goog) that maps a GKE cluster's Kubernetes service accounts into GCP's IAM system, enabling federated authentication without key files.

• KSA (Kubernetes Service Account): A namespace-scoped Kubernetes identity assigned to pods via the serviceAccountName field in the pod spec, annotated with the target GCP service account email for Workload Identity binding.

• GSA (GCP Service Account): A GCP IAM identity with specific role bindings that defines what GCP resources a workload can access, bound to one or more KSAs through the roles/iam.workloadIdentityUser role.

• GKE Metadata Server: A node-level daemon that intercepts pod requests to the metadata endpoint (169.254.169.254), authenticates the requesting pod's KSA, and returns short-lived GCP access tokens scoped to the bound GSA.

• Credential Scope Boundary: The intersection of a GSA's IAM roles and the Kubernetes namespace boundary that together define the maximum privilege envelope for any pod using Workload Identity — enforced independently by both GCP IAM and Kubernetes RBAC.

Concepts

GKE Workload Identity replaces static service-account keys by having the node-level metadata server exchange a pod's Kubernetes service account identity for short-lived, GSA-scoped GCP tokens, with the resulting privilege envelope bounded jointly by GCP IAM roles and the Kubernetes namespace.

Integration with the Broader GKE Security Posture

Workload Identity does not operate in isolation — it forms the authentication layer that other security controls depend upon. Pod Security Admission policies should enforce that pods in AI namespaces cannot mount hostPath volumes or use hostNetwork, which could bypass the GKE metadata server interception. Network policies should restrict egress to only the GCP APIs each namespace needs — a training namespace might need storage.googleapis.com and aiplatform.googleapis.com, while a serving namespace needs secretmanager.googleapis.com. Falco runtime rules should alert on any process inside an AI workload pod that attempts to read files matching the KEY_FILE_PATTERNS — this catches both residual key files from pre-migration configurations and active exfiltration attempts. Binary Authorization verifies that only attested container images can run in the cluster, which means that even if an attacker obtains a Workload Identity token, they cannot deploy a malicious container to use it.

The compliance scan output integrates with these layers: when detect_key_based_auth finds violations, the remediation path involves updating the pod spec to remove the mounted secret volume, updating the deployment's serviceAccountName to reference the annotated KSA, and verifying that the pod successfully authenticates through the GKE metadata server. This migration — from key-based to Workload Identity authentication — is the single highest-impact security improvement most AI platform teams can make, eliminating the entire class of credential exfiltration attacks that have historically been the primary vector for cloud-native breaches targeting machine learning infrastructure.

Code Walkthrough

Building on the Workload Identity Pool and KSA/GSA binding model from the Concepts section, the following implementation shows how to programmatically establish the full trust chain in a production AI platform.

The WorkloadIdentityManager class orchestrates GSA creation, the roles/iam.workloadIdentityUser IAM binding, and the iam.gke.io/gcp-service-account KSA annotation in a single call. The setup_binding method separates GCP-side operations from Kubernetes-side operations so that cluster administrators and GCP project administrators can own their respective surfaces independently.

Code snippetpython
1from google.cloud import iam_admin_v1
2from google.iam.v1 import iam_policy_pb2, policy_pb2
3from kubernetes import client as k8s_client, config as k8s_config
4import logging
5
6logger = logging.getLogger(__name__)
7
8class WorkloadIdentityManager:
9    def __init__(self, project_id: str, cluster_name: str):
10        self.project_id = project_id
11        self.cluster_name = cluster_name
12        self.wi_pool = f"{project_id}.svc.id.goog"
13        self.iam_client = iam_admin_v1.IAMClient()
14        k8s_config.load_incluster_config()
15        self.k8s_core = k8s_client.CoreV1Api()
16
17    def setup_binding(self, namespace: str, ksa_name: str,
18                      gsa_name: str) -> dict:
19        gsa_email = f"{gsa_name}@{self.project_id}.iam.gserviceaccount.com"
20        self._ensure_gsa_exists(gsa_name, gsa_email)
21        self._bind_ksa_to_gsa(gsa_email, namespace, ksa_name)
22        self._annotate_ksa(namespace, ksa_name, gsa_email)
23        logger.info("Binding complete: %s/%s -> %s", namespace, ksa_name, gsa_email)
24        return {"gsa_email": gsa_email,
25                "ksa": f"{namespace}/{ksa_name}",
26                "wi_pool": self.wi_pool}
27
28    def _ensure_gsa_exists(self, name: str, email: str):
29        request = iam_admin_v1.CreateServiceAccountRequest(
30            name=f"projects/{self.project_id}",
31            account_id=name,
32            service_account=iam_admin_v1.ServiceAccount(
33                display_name=f"WI-managed: {name}"))
34        try:
35            return self.iam_client.create_service_account(request=request)
36        except Exception as exc:
37            if "alreadyExists" in str(exc):
38                logger.info("GSA %s already exists, reusing", email)
39                return None
40            raise
41
42    def _bind_ksa_to_gsa(self, gsa_email: str, namespace: str, ksa_name: str):
43        resource = f"projects/{self.project_id}/serviceAccounts/{gsa_email}"
44        member = f"serviceAccount:{self.wi_pool}[{namespace}/{ksa_name}]"
45        policy = self.iam_client.get_iam_policy(
46            request=iam_policy_pb2.GetIamPolicyRequest(resource=resource))
47        policy.bindings.append(policy_pb2.Binding(
48            role="roles/iam.workloadIdentityUser", members=[member]))
49        self.iam_client.set_iam_policy(
50            request=iam_policy_pb2.SetIamPolicyRequest(
51                resource=resource, policy=policy))
52
53    def _annotate_ksa(self, namespace: str, ksa_name: str, gsa_email: str):
54        self.k8s_core.patch_namespaced_service_account(
55            name=ksa_name, namespace=namespace,
56            body={"metadata": {"annotations": {
57                "iam.gke.io/gcp-service-account": gsa_email}}})

Once setup_binding completes, the GKE metadata server can exchange the pod's KSA identity for a short-lived token on every GCP API call. To confirm the chain is active from inside the pod, query the metadata endpoint and compare the returned email against the expected GSA.

Code snippetpython
1import requests
2
3def verify_workload_identity(expected_gsa_email: str) -> bool:
4    """Return True when the pod authenticates via Workload Identity."""
5    url = (
6        "http://metadata.google.internal/computeMetadata/v1"
7        "/instance/service-accounts/default/email"
8    )
9    try:
10        response = requests.get(
11            url, headers={"Metadata-Flavor": "Google"}, timeout=5)
12        response.raise_for_status()
13        actual = response.text.strip()
14        match = (actual == expected_gsa_email)
15        logger.info("WI check — expected=%s actual=%s match=%s",
16                    expected_gsa_email, actual, match)
17        return match
18    except requests.RequestException as exc:
19        logger.error("Metadata endpoint unreachable: %s", exc)
20        return False

Check that verify_workload_identity returns True when called from inside the pod, and that no mounted key file appears in the pod's volume list — both together confirm that the full KSA-to-GSA trust chain is active and the pod carries no persistent credential material.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do grant roles/iam.workloadIdentityUser on the GSA's IAM policy with the full Workload Identity Pool member string (serviceAccount:{project}.svc.id.goog[{namespace}/{ksa_name}]) — this is the binding that authorizes the GKE metadata server to exchange KSA tokens for short-lived GSA credentials, and an incorrect member string silently produces a valid-looking setup that always returns 403 at runtime.
2. ✓Do annotate the Kubernetes service account with iam.gke.io/gcp-service-account pointing to the GSA email after the IAM binding is in place — the KSA annotation is what tells the GKE metadata server which GSA identity to impersonate, and pods scheduled before the annotation is applied will fall back to the node's default service account instead.
3. ✓Do verify the trust chain from inside the pod by querying http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email with the Metadata-Flavor: Google header and confirming the returned email matches the expected GSA — and confirm no key file volume is mounted — because both checks together are the only reliable signal that the pod is using short-lived federated tokens rather than a silently-retained JSON key.

Don'ts

1. ✗Don't mount a JSON service account key as a Kubernetes Secret to grant pods GCP access — downloaded key files are long-lived (up to ten years), exist entirely outside GCP's IAM revocation graph once written to disk, and a single compromised pod exposes credentials usable from anywhere on the internet, which is precisely the attack surface Workload Identity is designed to eliminate.
2. ✗Don't skip the _ensure_gsa_exists idempotency check before calling create_service_account — attempting to create a GSA that already exists raises an exception that aborts the entire setup_binding call; catching "alreadyExists" and reusing the existing GSA lets the binding logic run safely in redeployment and reconciliation loops without manual cleanup.
3. ✗Don't apply the roles/iam.workloadIdentityUser binding at the GCP project level instead of scoping it to the specific GSA resource — a project-level binding grants every KSA in the pool the ability to impersonate every GSA in the project, violating least-privilege and undermining the per-workload isolation that _bind_ksa_to_gsa establishes by targeting projects/{project}/serviceAccounts/{gsa_email}.