Deploy LLM API gateway on GKE with LiteLLM

Introduction

When you run LiteLLM locally it's a productivity shortcut; when it handles every authenticated LLM call in production, it becomes the trust boundary your security posture depends on. Deploying it on GKE means wiring together a hardened pod spec, zone-aware scheduling, Workload Identity for provider secrets, mTLS between services, and the Redis and Postgres companions that hold rate-limit state and audit logs. By the end of this lesson you'll have a production-grade LiteLLM gateway running on GKE — three replicas spread across availability zones, provider API keys synced from Secret Manager, and a rollout strategy that ships new policy without dropping in-flight requests.

Key Terminology

• topologySpreadConstraints — A Kubernetes scheduling directive that distributes pods across failure domains; the lesson configures maxSkew: 1 with topologyKey: topology.kubernetes.io/zone so each of the three replicas lands in a different availability zone, limiting the blast radius of a zonal outage to one pod.
• External Secrets Operator (ESO) — A Kubernetes controller that syncs secrets from an external vault into in-cluster Secrets on a configurable interval; the lesson uses a 30-minute refreshInterval against Google Secret Manager so provider keys such as openai_api_key propagate to running pods without requiring a rolling restart on each rotation cycle.
• PodDisruptionBudget (PDB) — A policy object that caps involuntary pod removals; setting minAvailable: 2 across three replicas ensures node drains and cluster upgrades can never reduce the LiteLLM gateway below two serving instances.
• Digest-pinned image — Referencing a container image by its SHA256 digest (e.g., ghcr.io/berriai/litellm@sha256:<pinned-digest>) rather than a mutable tag, so the exact binary promoted from staging can never be silently replaced by an upstream update.
• PSA restricted profile — Pod Security Admission enforcement applied at the ai-llm-gateway namespace level that mandates runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation: false, and capabilities: {drop: [ALL]} on every pod admitted to the namespace.
• maxUnavailable: 0 rolling update — A RollingUpdate strategy configuration that prevents any existing pod from being terminated before its replacement passes readiness, ensuring in-flight LLM requests are never dropped when a new gateway image or policy configuration is rolled out.

Concepts

The Gateway as a Trust Boundary

When LiteLLM runs locally it is a developer convenience; promoted to GKE it becomes the single choke point through which every authenticated LLM call passes. That architectural shift changes what "correctness" means. A misconfigured firewall rule, a provider key exposed in etcd, or replicas accidentally colocated in one zone are not inconveniences — they are production security events. The manifests in this lesson encode those concerns as enforceable Kubernetes primitives rather than operator runbooks: the namespace's PSA restricted profile blocks privilege escalation at admission time, a NetworkPolicy constrains ingress to the GCLB and AI workload namespaces, and Istio STRICT mTLS encrypts every byte of in-cluster traffic between the gateway, Redis, and Postgres. The lesson's security context — runAsUser: 65534, readOnlyRootFilesystem: true, capabilities: {drop: [ALL]} — expresses the same philosophy at the container level (see Code Walkthrough).

Zone Redundancy Requires Both Scheduling Constraints and Rollout Discipline

Three replicas alone do not buy zone redundancy — the scheduler can legitimately land all three pods on nodes inside the same availability zone. The topologySpreadConstraints block is the directive that overrides that default: maxSkew: 1 with topologyKey: topology.kubernetes.io/zone forces the scheduler to spread pods across zones so a zonal failure takes at most one pod offline. That covers the steady state. During a rollout, maxUnavailable: 0 closes a second gap: no old pod is evicted before its replacement clears the readiness probe, so in-flight LLM requests are never dropped when a new image or policy ships. The PodDisruptionBudget (minAvailable: 2) closes the third gap — voluntary disruptions such as node drains during cluster upgrades cannot collapse the gateway below two serving replicas even when the scheduler and rollout controller are not involved (see Code Walkthrough).

Decoupling Secrets Rotation from Pod Restarts

Provider API keys carry their own rotation cadence and should not be coupled to pod lifecycle events. Storing them as plain Kubernetes Secrets exposes key material to etcd if encryption-at-rest is misconfigured, and rotating them the naïve way — update the Secret, trigger a rolling restart — creates an unnecessary service disruption window. External Secrets Operator breaks both couplings: Google Secret Manager is the authoritative store; the ExternalSecret object's 30-minute refreshInterval propagates a rotated key into the in-cluster Secret automatically, and the pod mounts that Secret into a tmpfs volume so the material is never written to node disk. When quarterly rotation updates the value in Secret Manager, the in-cluster copy refreshes within half an hour and the running LiteLLM process picks up the new credentials on its next connection — no rolling restart required, no audit-log gap.

Code Walkthrough

Now that you understand the Redis and Postgres companion roles and the pitfalls of skipping topology spread or storing provider keys in plain Kubernetes Secrets, the manifests below wire these principles into a deployable unit.

The Hardened Deployment

The gateway runs in a dedicated ai-llm-gateway namespace under PSA restricted. Three replicas, a topologySpreadConstraints block, and a RollingUpdate strategy with maxUnavailable: 0 are the three levers that turn a single pod into a zone-redundant, zero-downtime gateway.

Code snippetyaml
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: litellm
5  namespace: ai-llm-gateway
6spec:
7  replicas: 3
8  strategy:
9    type: RollingUpdate
10    rollingUpdate: {maxSurge: 1, maxUnavailable: 0}
11  template:
12    metadata:
13      labels: {app: litellm}
14    spec:
15      serviceAccountName: litellm-sa
16      topologySpreadConstraints:
17      - maxSkew: 1
18        topologyKey: topology.kubernetes.io/zone
19        whenUnsatisfiable: ScheduleAnyway
20        labelSelector: {matchLabels: {app: litellm}}
21      containers:
22      - name: litellm
23        image: ghcr.io/berriai/litellm@sha256:<pinned-digest>
24        securityContext:
25          runAsNonRoot: true
26          runAsUser: 65534
27          readOnlyRootFilesystem: true
28          allowPrivilegeEscalation: false
29          capabilities: {drop: [ALL]}
30        ports: [{name: https, containerPort: 4000}]
31        env:
32        - name: LITELLM_MASTER_KEY
33          valueFrom: {secretKeyRef: {name: litellm-master-key, key: key}}
34        - name: DATABASE_URL
35          valueFrom: {secretKeyRef: {name: litellm-db-url, key: url}}
36        - name: REDIS_URL
37          value: rediss://redis.ai-llm-gateway.svc:6379
38        livenessProbe:
39          httpGet: {path: /health/liveliness, port: https, scheme: HTTPS}
40        readinessProbe:
41          httpGet: {path: /health/readiness, port: https, scheme: HTTPS}
42        resources:
43          requests: {cpu: "1", memory: 1Gi}
44          limits:   {cpu: "4", memory: 4Gi}

topologySpreadConstraints with maxSkew: 1 ensures replicas land in different zones — a zonal outage takes at most one of the three pods. The image is pinned by digest, matching the operating discipline of tracking upstream releases on a delay and vetting in staging before promoting. REDIS_URL uses the rediss:// (TLS) scheme to reach the Redis companion that holds 24-hour rate-limit state; DATABASE_URL connects to the Postgres companion for spend tracking and audit logs.

Provider Credentials via External Secrets

Provider API keys must not live as plain text in a Kubernetes Secret — the etcd-encryption misconfiguration pitfall is real. External Secrets Operator syncs them from Google Secret Manager on a 30-minute interval, keeping the in-cluster copy fresh without pod restarts on every rotation cycle.

Code snippetyaml
1apiVersion: external-secrets.io/v1beta1
2kind: ExternalSecret
3metadata:
4  name: provider-keys
5  namespace: ai-llm-gateway
6spec:
7  refreshInterval: 30m
8  secretStoreRef:
9    kind: ClusterSecretStore
10    name: gsm-cluster-store
11  target:
12    name: litellm-provider-keys
13    creationPolicy: Owner
14  data:
15  - secretKey: openai_api_key
16    remoteRef: {key: providers/openai-api-key, version: latest}
17  - secretKey: anthropic_api_key
18    remoteRef: {key: providers/anthropic-api-key, version: latest}
19  - secretKey: gemini_api_key
20    remoteRef: {key: providers/gemini-api-key, version: latest}

The synced Secret is mounted as tmpfs in the pod and never written to disk. Quarterly key rotation only requires updating the value in Secret Manager; the 30-minute refresh propagates the new key to running pods without a rolling restart.

A PodDisruptionBudget with minAvailable: 2 across the three replicas prevents node drains from ever dropping below two serving pods. In-cluster traffic between the gateway, Redis, and Postgres runs over Istio STRICT mTLS, and a NetworkPolicy restricts ingress to the GCLB and AI workload namespaces — keeping the audit Postgres isolated from application workload bursts.

Confirm that kubectl rollout status deployment/litellm -n ai-llm-gateway reports successfully rolled out, all three pods appear in distinct zones when you inspect the topology.kubernetes.io/zone node label, and a test request through the GCLB endpoint returns a response containing the x-litellm-version header with no upstream error code.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do pin the LiteLLM container image by SHA256 digest — a mutable tag like latest lets an upstream release slip into production without staging vetting; digest pinning ensures the exact binary you tested is the one that runs across all three replicas.
2. ✓Do set maxUnavailable: 0 in the RollingUpdate strategy and pair it with a PodDisruptionBudget of minAvailable: 2 — together these two controls guarantee the gateway never drops below two serving replicas during a rollout or node drain, preserving rate-limit continuity backed by the shared Redis state.
3. ✓Do sync provider API keys from Google Secret Manager via External Secrets Operator with a refreshInterval: 30m — this keeps the in-cluster Secret fresh on key rotation without pod restarts, and avoids writing plaintext credentials to etcd where an encryption misconfiguration would expose them.

Don'ts

1. ✗Don't store provider API keys as plain Kubernetes Secrets created manually — etcd encryption is often misconfigured in practice, and a plain Secret means a compromised cluster read is a full credential leak; the ExternalSecret → GSM path keeps the authoritative value outside the cluster entirely.
2. ✗Don't omit topologySpreadConstraints with topologyKey: topology.kubernetes.io/zone — without it, the Kubernetes scheduler can place all three LiteLLM replicas in the same availability zone, turning a single zonal outage into a complete gateway outage despite the replica count.
3. ✗Don't use the redis:// scheme for the REDIS_URL environment variable — plaintext Redis traffic exposes rate-limit tokens and session state on the cluster network; the manifest uses rediss:// (TLS) to reach the Redis companion, and dropping the second s silently downgrades the connection without any startup error.