Design agent execution model with sandboxed pods

Introduction

When you deploy agents at scale, a single rogue process can exhaust shared resources, leak credentials, or degrade every co-located workload. Without explicit isolation boundaries, one agent's memory leak becomes every tenant's latency problem. The sandboxed pod model solves this by mapping each agent invocation to exactly one Kubernetes pod with dedicated CPU, memory, and network controls, so the blast radius of any failure stays confined to that single run. By the end of this lesson, you'll be able to design the execution request model, translate it into a pod manifest with security contexts and resource limits, and wire the four components—submission, spec building, scheduling, and lifecycle control—into a coherent agent runtime.

Key Terminology

• Agent Runtime: The platform subsystem responsible for accepting agent execution requests, creating isolated execution environments, managing lifecycle state, and collecting outputs upon completion.

• Sandboxed Pod: A Kubernetes pod configured with strict resource limits, security contexts, and network policies that confine an agent's execution to prevent interference with other workloads or the host system.

• Execution Model: The architectural blueprint that defines how an agent run maps to infrastructure primitives—in this design, one agent run equals one Kubernetes pod with dedicated resources and a finite lifetime.

• Blast Radius: The scope of impact when an agent fails catastrophically. Pod-level isolation ensures the blast radius is limited to a single agent run, leaving other pods and the control plane unaffected.

• Security Context: A Kubernetes configuration block that controls process-level security settings including user ID, group ID, capability sets, privilege escalation policies, and filesystem access modes.

• emptyDir Volume: A Kubernetes volume type that provides ephemeral storage backed by the node's disk or memory. It exists for the lifetime of the pod and is deleted when the pod terminates, making it ideal for agent workspace scratch space.

• activeDeadlineSeconds: A Kubernetes pod spec field that sets a hard wall-clock timeout for the entire pod. When the deadline expires, Kubernetes terminates the pod regardless of container state, serving as the ultimate safeguard against runaway agents.

• Resource Requests vs. Limits: Requests tell the Kubernetes scheduler how much capacity a pod needs for placement decisions. Limits enforce hard ceilings enforced by the kernel via cgroups—exceeding a memory limit triggers OOMKill, while exceeding a CPU limit triggers throttling.

Concepts

Why Sandboxed Pods for Agent Execution

Agents are inherently unpredictable. Unlike traditional microservices that handle well-defined request/response cycles, an agent may execute arbitrary tool calls, spawn subprocesses, download dependencies, or consume unbounded memory while reasoning over large contexts. Running multiple agents inside a shared process or even a shared container creates cascading failure modes: one agent's memory leak starves another, one agent's network call to a malicious endpoint exposes shared credentials, and one agent's CPU-bound loop degrades latency for all co-located workloads.

The sandboxed pod model solves these problems by mapping each agent execution to exactly one Kubernetes pod. This gives you process-level isolation via Linux namespaces, resource limits via cgroups, network segmentation via network policies, and filesystem isolation via per-pod volumes. The pod becomes the blast radius boundary—when an agent misbehaves, Kubernetes kills that single pod without affecting any other running agents.

Code Walkthrough

Now that you understand why each agent run needs its own isolated environment, let's trace how a request moves through the runtime's four layers and what data structures anchor each step.

When a client posts to POST /agent-runs, the submission layer validates and enqueues the request. The scheduler dequeues it, checks cluster capacity, and delegates to the spec builder, which translates the request into a Kubernetes pod manifest. Once the pod runs, the lifecycle controller watches its state transitions and writes domain-level execution states to the state store—shielding clients from raw Kubernetes pod phases.

Before the spec builder can construct a pod manifest, it needs a fully validated execution request. The AgentExecutionRequest dataclass captures the agent image, entry command, environment variables, and a nested ExecutionSandboxConfig that declares CPU and memory limits, an activeDeadlineSeconds-equivalent timeout, and the AgentNetworkPolicy controlling egress. The validate method enforces invariants that Kubernetes would only surface at pod creation time—catching misconfiguration early, before any cluster resource is consumed.

Code snippetpython
1from dataclasses import dataclass, field
2from enum import Enum
3
4class AgentNetworkPolicy(Enum):
5    NONE = "none"
6    EGRESS_RESTRICTED = "egress_restricted"
7    EGRESS_ALLOW_LIST = "egress_allow_list"
8    FULL = "full"
9
10@dataclass
11class ExecutionSandboxConfig:
12    cpu_limit: str = "2000m"
13    cpu_request: str = "500m"
14    memory_limit: str = "2Gi"
15    memory_request: str = "512Mi"
16    timeout_seconds: int = 600
17    max_ephemeral_storage: str = "5Gi"
18    network_policy: AgentNetworkPolicy = AgentNetworkPolicy.EGRESS_RESTRICTED
19    allowed_egress_cidrs: list[str] = field(default_factory=list)
20    read_only_root_fs: bool = True
21    run_as_non_root: bool = True
22    drop_all_capabilities: bool = True
23
24@dataclass
25class AgentExecutionRequest:
26    agent_image: str
27    agent_id: str
28    run_id: str
29    tenant_id: str
30    entry_command: list[str]
31    env_vars: dict[str, str] = field(default_factory=dict)
32    sandbox: ExecutionSandboxConfig = field(
33        default_factory=ExecutionSandboxConfig
34    )
35    workspace_size: str = "10Gi"
36    priority: int = 10
37    labels: dict[str, str] = field(default_factory=dict)
38
39    def validate(self) -> None:
40        if not self.agent_image:
41            raise ValueError("agent_image is required")
42        if not self.run_id:
43            raise ValueError("run_id is required")
44        if self.sandbox.timeout_seconds <= 0:
45            raise ValueError("timeout_seconds must be positive")
46        if self.priority not in range(1, 11):
47            raise ValueError("priority must be between 1 and 10")

The ExecutionSandboxConfig defaults encode the blast-radius principle directly in code. Two CPU cores as the upper bound, two gigabytes of memory, a ten-minute hard timeout, egress restricted by default, a read-only root filesystem, non-root execution, and all Linux capabilities dropped—any field left at its default produces a pod that cannot escalate privileges, write to the container filesystem, or reach arbitrary external endpoints. These are the security context settings the spec builder will later project into the pod manifest's securityContext block, making the runtime's safety guarantees explicit and auditable at the request layer rather than buried in infrastructure configuration.

Confirm that instantiating AgentExecutionRequest with a blank agent_image raises ValueError before any Kubernetes API call is made.

Do's and Don'ts

Do's

1. ✓

   Do set both resource requests and limits on every agent pod — Requests without limits allow unbounded consumption; limits without requests cause poor scheduling decisions. Always specify both to get predictable placement and hard enforcement. A pod with requests.memory=512Mi and limits.memory=2Gi gets scheduled on nodes with 512Mi available but can burst up to 2Gi before getting OOMKilled.

2. ✓

   Do use activeDeadlineSeconds as a hard backstop — In-process timeouts can be bypassed by blocked I/O or infinite loops in native code. The Kubernetes-level deadline is enforced by the kubelet externally, guaranteeing that no pod runs beyond its allotted time regardless of what the agent process does internally.

3. ✓

   Do mount a dedicated /tmp volume when using read-only root filesystems — Many Python packages, serialization libraries, and ML frameworks write to /tmp during normal operation. Without an explicit writable mount at /tmp, agents will crash with permission errors that are difficult to diagnose in production.

Don'ts

1. ✗

   Don't run agent containers as root — Even with capability dropping and read-only filesystems, running as UID 0 exposes you to container escape vulnerabilities that exploit root-only kernel paths. Always set runAsUser and runAsGroup to a non-root UID, and set runAsNonRoot to True at the pod level so Kubernetes rejects any image that defaults to root.

2. ✗

   Don't share workspace volumes across agent pods — Sharing a PersistentVolumeClaim between concurrent agents creates data races, accidental file overwrites, and cross-agent information leakage. Each agent gets its own emptyDir that is created fresh and destroyed on termination.

3. ✗

   Don't set restartPolicy to "Always" or "OnFailure" for agent pods — Agent runs are one-shot operations. Automatic restarts mask failures, produce duplicate side effects (duplicate API calls, duplicate file writes), and complicate execution state tracking. Set the policy to "Never" and let the platform's scheduler handle retry logic at a higher level with full visibility into why the previous run failed.