Deploy agent security monitor on GKE

Introduction

In production, deploying a security sidecar alongside your agent runtime means separating the security team's release cadence from the agent team's — but hard-coding environment URLs in container images defeats that flexibility. When you manage the sidecar's kill-switch endpoint, egress allowlist, and resource limits through a Helm values.yaml, each environment gets its own overrides without a rebuild. By the end of this lesson, you'll be able to author a production-ready values.yaml that pins the security sidecar by image digest, configures the kill-switch URL via chart values rather than the image, enforces a pod disruption budget, and restricts egress to the LLM gateway and tool registry.

Key Terminology

• Image digest pin — a reference to the sidecar container image by its immutable sha256: content address (e.g., registry.example.com/agents/security@sha256:a7c3f1e8b2d94c0fa...), guaranteeing that exactly the audited binary is deployed rather than whatever a mutable tag currently points to.
• Values overlay — a secondary Helm values file (e.g., values-prod.yaml) that overrides only the fields that differ per environment; merged at deploy time via multiple -f flags in helm upgrade --install, keeping the base values.yaml environment-agnostic and rebuild-free.
• Kill-switch URL (killSwitchUrl) — the sidecar.env.killSwitchUrl chart value that tells the security sidecar which in-cluster endpoint to call to halt agent execution; sourced from chart values rather than baked into the image so it resolves to the correct service in every environment without a container rebuild.
• PodDisruptionBudget (pdb.minAvailable) — a Kubernetes policy, driven by the pdb.minAvailable chart value, that caps how many agent pods the cluster may take offline simultaneously during node-pool upgrades or voluntary disruptions, preserving continuous coverage from the security monitor fleet.
• Egress allowlist — the NetworkPolicy rendered by the chart that restricts the sidecar's outbound connections to the namespaces and pod labels declared in network.llmGatewayNamespace and network.toolRegistryPodLabel, bounding the blast radius of any tool-misuse attempt.
• forbidPatterns — the list of pattern identifiers (e.g., sql_meta, shell_exfil, protocol_smuggle) in sidecar.env that the security sidecar evaluates against agent tool calls at runtime to detect and block prohibited request shapes before they reach downstream systems.

Concepts

Decoupling the Security Sidecar's Release Cadence

The central design idea behind this lesson's values.yaml is that the security team and the agent team ship on independent schedules. If the kill-switch URL, sidecar binary version, or egress rules were baked into the agent image, a security hotfix would require the agent team to cut a new release — slowing incident response at exactly the moment speed matters most. By externalizing every security-team-owned decision into Helm chart values, the security team controls a file they can change, PR-review, and deploy without touching the agent image at all.

This separation is reinforced structurally by the chart linter: if sidecar.resources.limits is absent, CI rejects the rollout before any manifest reaches the cluster. The linter acts as a policy gate — encoding the operational rule that a sidecar without CPU and memory limits can starve the agent runtime it is supposed to protect.

Image Digest Pinning and Configuration Externalization

Helm charts can reference images by tag or by digest. Tags are mutable — even a semver tag can be silently overwritten in a registry. A sha256: digest is an immutable content address: the exact bytes deployed are known, reproducible, and auditable against a signed artifact. For a security sidecar — the component whose entire job is detecting adversarial behavior — a mutable image reference undermines the trustworthiness of the defense plane itself.

The same immutability logic applies to environment-specific configuration. The killSwitchUrl differs between staging and production; if it were a ENV instruction in the Dockerfile, promoting the image across environments would either silently call the wrong endpoint or require a separate image build per environment. Sourcing it from sidecar.env.killSwitchUrl means the same digest runs everywhere while each environment resolves to its correct kill-switch service (see Code Walkthrough).

Defense-Plane Availability: PDB and Rollout Strategy

A security sidecar that goes dark during a Kubernetes node-pool upgrade leaves agents unmonitored during that window. The pdb.minAvailable: "80%" value instructs Kubernetes to drain at most 20 % of agent pods at a time during voluntary disruptions — keeping the majority of the security monitor fleet running continuously. Paired with rollout.maxUnavailable: 0, the chart's Deployment template ensures no pod is terminated before a healthy replacement is ready.

Both values belong in values.yaml — not hard-coded in the Deployment template — because different environments carry different risk tolerances. A development cluster might accept a lower minAvailable for faster node recycling; production holds at 80 %. Capturing that decision in values-prod.yaml places it in a version-controlled, PR-reviewed file alongside all other production configuration, rather than buried in a shared template that every environment shares without differentiation.

Code Walkthrough

Building on the operating discipline from the Concepts section, the values.yaml is the single file the security team edits before each rollout — image digest, kill-switch URL, resource limits, egress targets, and disruption tolerance all live here rather than inside the container image.

Code snippetyaml
1# values.yaml  —  security-monitor Helm chart
2sidecar:
3  # Pin by digest; the security team controls exactly which binary runs.
4  image: registry.example.com/agents/security@sha256:a7c3f1e8b2d94c0fa631785e4d2b9c17
5  resources:
6    requests: {cpu: 200m, memory: 256Mi}
7    limits:   {cpu: 500m, memory: 512Mi}
8  env:
9    # Kill-switch URL sourced from values, never baked into the image.
10    killSwitchUrl: http://kill-switch.agents.svc:8081
11    baselineBucket: gs://agent-baselines/v3
12    forbidPatterns: [sql_meta, shell_exfil, protocol_smuggle]
13
14agent:
15  image: registry.example.com/agents/runtime:1.9.0
16  serviceAccount: agent-sa
17
18network:
19  llmGatewayNamespace: ai-llm-gateway
20  toolRegistryPodLabel: tool-registry
21
22pdb:
23  minAvailable: "80%"
24
25rollout:
26  maxSurge: 1
27  maxUnavailable: 0

A separate values-prod.yaml overlays only the fields that differ in production — typically the sidecar digest (a newer pinned hash) and the killSwitchUrl (the production endpoint). The chart then renders the Deployment, NetworkPolicy, ServiceAccount, and PodDisruptionBudget from a single invocation:

Code snippetbash
1helm upgrade --install agent-security ./chart \
2  -f values.yaml \
3  -f values-prod.yaml \
4  --namespace agents

The chart's linter rejects a missing sidecar.resources.limits block before the manifest reaches the cluster, enforcing a resource-limit requirement — a sidecar without limits can starve the agent runtime it is supposed to protect.

Because pdb.minAvailable and the rollout strategy are also chart values, the security team adjusts disruption tolerance per environment without touching the Deployment template. They edit values-prod.yaml, open a PR, and CI lints and deploys in one step — decoupling the security sidecar's release cadence from the agent's release cadence exactly as the Concepts section calls for.

Confirm that helm template ./chart -f values.yaml -f values-prod.yaml renders a Deployment whose sidecar container references a digest-pinned image, a PodDisruptionBudget with minAvailable: 80%, and no literal kill-switch hostname anywhere in the container's env block.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do pin sidecar.image by SHA-256 digest in values.yaml — a mutable tag like latest or 1.9.0 can be silently overwritten in the registry, letting an untested binary reach production without any chart change; the digest-pinned form registry.example.com/agents/security@sha256:a7c3f1e8b2d94c0fa631785e4d2b9c17 gives the security team deterministic control over exactly which binary runs in each environment.
2. ✓Do source sidecar.env.killSwitchUrl (and other environment-varying endpoints) from chart values, not from the container image — externalizing the endpoint into values.yaml and overriding it per environment in values-prod.yaml lets the security team change the kill-switch target without a rebuild, which is the mechanism that decouples the security team's release cadence from the agent team's.
3. ✓Do run helm template ./chart -f values.yaml -f values-prod.yaml before every helm upgrade — inspecting the rendered output is the only way to confirm the sidecar container references a digest-pinned image, the PodDisruptionBudget carries minAvailable: 80%, and no literal kill-switch hostname appears in the container's env block.

Don'ts

1. ✗Don't omit sidecar.resources.limits from values.yaml — the chart linter rejects a manifest missing this block before it reaches the cluster, but if the linter is bypassed, a limitless sidecar can exhaust node CPU and memory and starve the agent runtime it is supposed to protect; the limits: {cpu: 500m, memory: 512Mi} block is the guard against that failure mode.
2. ✗Don't put production-specific fields — the pinned sidecar digest and the production killSwitchUrl — in the base values.yaml — collapsing environment overrides into the base file erases the environment boundary, making it impossible to distinguish what changed between environments in a PR and defeating the layered helm upgrade -f values.yaml -f values-prod.yaml pattern the chart is built around.
3. ✗Don't leave network.llmGatewayNamespace or network.toolRegistryPodLabel unset or misnamed — both values feed the chart's NetworkPolicy template that restricts egress to the LLM gateway and tool registry; a wrong label selector renders a policy whose pod-selector matches nothing, silently leaving the sidecar with unrestricted egress and undoing the tool-misuse containment the chart is designed to enforce.