Deploy secure MCP infrastructure on GKE

Introduction

Engineers often build robust MCP security controls — tool poisoning detectors, OAuth2 authentication, A2A signing gateways, and cross-server exfiltration policies — only to deploy them into a shared namespace with permissive defaults, undoing that work at the infrastructure layer. Without a hardened Kubernetes environment, an attacker who compromises one pod can move laterally across tool servers or bypass network controls entirely. By the end of this lesson, you'll be able to deploy MCP gateway and tool server components on GKE with namespace isolation, Workload Identity bindings, strict mTLS enforcement, and egress network policies that together enforce the security guarantees the protocol controls above them depend on.

Key Terminology

• Pod Security Admission (PSA) — a Kubernetes admission controller that enforces the restricted pod security profile at the namespace level via labels like pod-security.kubernetes.io/enforce: restricted, ensuring every pod in mcp-platform runs non-root (runAsUser: 65534), mounts a read-only root filesystem, and drops all Linux capabilities.
• Workload Identity — a GKE mechanism that maps a Kubernetes ServiceAccount to a Google Cloud service account via the iam.gke.io/gcp-service-account annotation, exchanging that binding for short-lived GCP tokens at runtime so no JSON key files are stored on disk or in cluster Secrets.
• PeerAuthentication — an Istio resource that controls the mTLS mode for a namespace; setting mode: STRICT means every pod-to-pod connection within mcp-platform must present a mutually authenticated TLS certificate, rejecting plaintext and one-way-TLS connections entirely.
• AuthorizationPolicy — an Istio resource that restricts which source namespaces and SPIFFE principals can reach a specific workload; without a principal-scoped policy, STRICT mTLS alone still permits any cluster pod holding a valid certificate to connect to mcp-gateway.
• Service account blast-radius isolation — the practice of assigning each tool server its own Kubernetes ServiceAccount bound to a distinct GCP service account, so a compromise of one tool server cannot leverage another server's GCP IAM scope or credentials.
• Egress proxy — an explicit outbound traffic gateway through which all egress from mcp-platform must route; without it, a compromised gateway can open direct connections to arbitrary public IPs, creating an exfiltration side-channel that protocol-layer cross-server policies never see.

Concepts

Infrastructure Isolation Is a Security Boundary, Not Ops Hygiene

MCP security controls — tool poisoning detectors, OAuth2 tokens, A2A signing gateways, cross-server exfiltration policies — operate at the protocol layer. They protect against threats that arrive through legitimate, authenticated channels. But if an attacker compromises one pod in a permissive shared namespace, the protocol controls become irrelevant: the attacker already has internal network access to call other tool servers directly, sidestepping every gate above them.

The mcp-platform namespace, labeled to enforce Pod Security Admission at restricted, is the foundation that makes the protocol controls meaningful. Enforcing that every container runs non-root, uses a read-only root filesystem, and drops all Linux capabilities does not prevent an initial compromise, but it aggressively shrinks post-compromise capability: the attacker cannot write persistent payloads to disk, cannot escalate to root, and cannot exploit kernel interfaces that require elevated capabilities. Treating the namespace as a strict production tier — not a convenient shared space — is what the lesson's operating discipline formalizes.

mTLS Verification Is Not Authorization

STRICT mTLS across mcp-platform ensures every pod-to-pod connection is mutually authenticated using certificates issued by the Istio certificate authority. But "both sides hold valid certificates" is not the same as "this caller is permitted to reach this endpoint." Any workload in the cluster — a metrics scraper, a debug container, a compromised pod in default — can hold a valid Istio certificate. Under STRICT mTLS alone, all of them can reach the gateway.

The AuthorizationPolicy closes this gap by binding permission to a specific SPIFFE principal and source namespace: only cluster.local/ns/ai-app/sa/agent-sa from the ai-app namespace can connect to mcp-gateway. A pod in any other namespace is denied even when it presents a valid mTLS certificate. This distinction — certificate validity versus principal authorization — is the exact pitfall the lesson highlights: deploying STRICT mTLS without a narrowly scoped AuthorizationPolicy leaves the gateway reachable by any cluster workload (see Code Walkthrough).

Workload Identity Eliminates the Static Credential Problem

Each tool server runs under its own Kubernetes ServiceAccount annotated to a distinct Google service account. GKE's Workload Identity mechanism exchanges that annotation for short-lived GCP tokens automatically — no JSON key files are stored in the pod, in a Kubernetes Secret, or anywhere on disk. This removes two attack surfaces at once: there is no credential file to steal from a compromised container, and credential rotation is continuous rather than a quarterly manual operation.

The per-server isolation compounds the benefit. Because each tool server binds to a different GCP service account, a compromise of one server's pod cannot yield a credential that widens access to another server's data or GCP IAM permissions. The blast radius of any single tool-server compromise is bounded by the compromised account's IAM policy, not by the attacker's ability to locate and export a shared key (see Code Walkthrough).

Egress Control Closes the Exfiltration Path

Protocol-layer cross-server exfiltration policies govern what a tool server returns through legitimate MCP channels. They cannot stop a compromised gateway from opening a direct outbound TCP connection to an attacker-controlled IP — because that connection bypasses the MCP protocol layer entirely. Routing all egress through an explicit proxy creates a chokepoint where outbound connections are inspected against an allowlist, logged, and blocked if they do not match. The Code Walkthrough reinforces this by resolving secrets through in-cluster Vault DNS (vault.secrets.svc:8200) rather than any public endpoint: even credential lookups never require a direct internet route, so the attack surface for a compromised pod is further reduced before an egress network policy is even applied.

Code Walkthrough

Now that you understand the operating discipline and pitfalls for MCP infrastructure — treating the namespace as a production tier, routing all egress through an explicit proxy, and auditing AuthorizationPolicy entries regularly — the following manifests show how to wire those principles together in a GKE cluster.

The first block defines the mcp-platform namespace with Pod Security Admission set to restricted and the MCP gateway Deployment. The namespace label enforces that every pod runs non-root with a read-only root filesystem and no privilege escalation. The gateway exposes only an HTTPS port, resolves secrets through in-cluster Vault service DNS so it never needs a public internet route for credentials, and carries both liveness and readiness probes so rolling updates drain correctly before traffic shifts.

Code snippetyaml
1apiVersion: v1
2kind: Namespace
3metadata:
4  name: mcp-platform
5  labels:
6    pod-security.kubernetes.io/enforce: restricted
7    pod-security.kubernetes.io/audit:   restricted
8    workload: ai-security
9---
10apiVersion: apps/v1
11kind: Deployment
12metadata:
13  name: mcp-gateway
14  namespace: mcp-platform
15spec:
16  replicas: 3
17  selector:
18    matchLabels:
19      app: mcp-gateway
20  template:
21    metadata:
22      labels:
23        app: mcp-gateway
24    spec:
25      serviceAccountName: mcp-gateway-sa
26      securityContext:
27        runAsNonRoot: true
28        runAsUser: 65534
29        seccompProfile:
30          type: RuntimeDefault
31      containers:
32      - name: gateway
33        image: registry.example.com/mcp/gateway:1.6.0
34        securityContext:
35          readOnlyRootFilesystem: true
36          allowPrivilegeEscalation: false
37          capabilities:
38            drop: [ALL]
39        ports:
40        - {name: https,   containerPort: 8443}
41        - {name: metrics, containerPort: 9090}
42        env:
43        - {name: VAULT_ADDR,                  value: "https://vault.secrets.svc:8200"}
44        - {name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://otel.observability:4317"}
45        livenessProbe:
46          httpGet: {path: /healthz, port: https, scheme: HTTPS}
47        readinessProbe:
48          httpGet: {path: /readyz,  port: https, scheme: HTTPS}
49        resources:
50          requests: {cpu: 500m, memory: 512Mi}
51          limits:   {cpu: 2,    memory: 2Gi}

The second block pairs three resources: the gateway's Kubernetes ServiceAccount annotated for Workload Identity, a PeerAuthentication that enforces STRICT mTLS across the entire namespace, and an AuthorizationPolicy that narrows ingress to the agent-sa ServiceAccount in the ai-app namespace specifically. This combination directly addresses the pitfall of allowing the agent service account to reach the gateway from any namespace — without the principal restriction, any cluster pod holding a valid certificate could connect even under STRICT mTLS.

Code snippetyaml
1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  name: mcp-gateway-sa
5  namespace: mcp-platform
6  annotations:
7    iam.gke.io/gcp-service-account: mcp-gateway@PROJECT.iam.gserviceaccount.com
8---
9apiVersion: security.istio.io/v1
10kind: PeerAuthentication
11metadata:
12  name: mcp-mtls
13  namespace: mcp-platform
14spec:
15  mtls:
16    mode: STRICT
17---
18apiVersion: security.istio.io/v1
19kind: AuthorizationPolicy
20metadata:
21  name: mcp-gateway-allow-app
22  namespace: mcp-platform
23spec:
24  selector:
25    matchLabels:
26      app: mcp-gateway
27  rules:
28  - from:
29    - source:
30        namespaces: [ai-app]
31        principals: ["cluster.local/ns/ai-app/sa/agent-sa"]

Each tool server gets its own Kubernetes ServiceAccount annotated to a distinct Google service account, so a compromise of one tool server cannot escalate to another server's data scope. No JSON key files are stored on disk; GCP's Workload Identity token exchange handles credential rotation automatically.

Confirm that kubectl get peerauthentication -n mcp-platform reports STRICT mode and that kubectl auth can-i --as=system:serviceaccount:default:default get pods -n mcp-platform returns no — both results together verify that namespace isolation and mTLS enforcement are active before you promote this configuration to production.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do pair PeerAuthentication STRICT mTLS with an AuthorizationPolicy that pins both namespaces and principals — STRICT mTLS alone authenticates the TLS certificate but permits any cluster pod holding a valid Istio-issued cert to reach the MCP gateway; restricting the source to cluster.local/ns/ai-app/sa/agent-sa is what closes the lateral-movement path that tool poisoning or a compromised sidecar could exploit.
2. ✓Do annotate each tool server's Kubernetes ServiceAccount with a distinct GCP service account via iam.gke.io/gcp-service-account — Workload Identity scopes every tool server's GCP permissions independently so that compromising one pod cannot escalate to another server's data or APIs; without per-server bindings, a single credential covers the entire mcp-platform namespace.
3. ✓Do label the mcp-platform namespace with pod-security.kubernetes.io/enforce: restricted — this admission-time control enforces runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation: false, and capabilities: drop: [ALL] before any pod lands, ensuring a misconfigured Deployment spec cannot silently bypass the pod-level security context declared in the gateway manifest.

Don'ts

1. ✗Don't treat STRICT mTLS as a complete authorization boundary without an AuthorizationPolicy — PeerAuthentication: STRICT only enforces mutual TLS handshakes; without the rules.from.source.principals restriction, any workload in the cluster that obtains a valid certificate (including default service accounts in unrelated namespaces) can send requests to the MCP gateway, which is the cross-server exfiltration vector the architecture is meant to eliminate.
2. ✗Don't store GCP service account JSON key files in the deployment instead of using Workload Identity annotations — static key files do not rotate, survive pod deletion as Kubernetes Secrets or mounted volumes, and grant persistent access across all tool servers if any one secret is exfiltrated; Workload Identity's short-lived token exchange is what makes a single-pod compromise non-escalating.
3. ✗Don't promote the mcp-platform configuration to production without confirming kubectl get peerauthentication -n mcp-platform shows STRICT and kubectl auth can-i --as=system:serviceaccount:default:default get pods -n mcp-platform returns no — a PeerAuthentication that silently falls back to PERMISSIVE mode (common when Istio injection is absent on a pod) produces a manifest that looks correct but leaves the namespace open to unauthenticated connections from any peer.