Deploy agent runtime with K8s Job controller

Introduction

When you deploy an agent runtime to Kubernetes, you need to coordinate a Job controller, a durable message queue, a log streaming DaemonSet, and a Helm chart that packages all of it together—getting any one of these wrong means lost work, resource contention, or invisible failures in production. Backpressure, RBAC bindings, ResourceQuota sizing, and Redis durability settings are not independent concerns; they compose into a single deployable surface that must be validated as a unit before any agent workload runs. By the end of this lesson, you will be able to configure each component of the agent runtime platform, wire backpressure between the queue consumer and the Job controller, and validate the release with a Helm test hook.

Key Terminology

• Job Controller: A custom Kubernetes controller that translates agent execution requests into K8s Job objects, watches their lifecycle, and enforces platform policies like resource limits and execution timeouts
• Helm Chart: A package format for Kubernetes that bundles all manifests, default values, and template logic into a versioned, installable unit
• Queue Consumer: A long-running process that dequeues agent work items from Redis and dispatches them to the Job Controller for execution
• Log Streamer: A DaemonSet-based component that tails container logs from agent pods and forwards them to an external storage backend
• Backpressure: A flow-control mechanism where the queue consumer slows its dequeue rate when the cluster cannot accept additional agent Jobs, preventing resource exhaustion
• RBAC Binding: A Kubernetes RoleBinding that grants the Job Controller service account permission to create, watch, and delete Job objects within the agent namespace

Concepts

The diagram below shows how the four runtime components packaged in the Helm chart interact at request time, so the budget, durability, and streaming concepts that follow have a shared mental picture to attach to.

Resource Budgets and Namespace Isolation

The Helm chart should define a ResourceQuota capping total CPU and memory for agent Jobs. A typical quota allocates 80% to agent pods and reserves 20% for platform components. Calibrate max_concurrent_jobs against this quota — if each pod requests 2 CPU and 4Gi, and the quota allows 40 CPU, cap at 20 concurrent jobs.

Queue Durability and At-Least-Once Delivery

Deploy Redis as a StatefulSet with a PersistentVolumeClaim and appendonly yes. The queue consumer should use BRPOPLPUSH (or XREADGROUP) to move items to a processing list before acknowledging them, with a reaper process re-enqueuing items stuck in processing beyond a timeout.

Log Streamer Deployment Strategy

The DaemonSet-based log streamer watches for pods with the label agent-runtime.io/managed=true. Production deployments typically read directly from /var/log/containers/ (lower latency, survives kubelet restarts) with a read-only hostPath mount and a SecurityContext dropping all capabilities except DAC_READ_SEARCH.

Helm Chart Versioning and Upgrade Safety

Follow strict semver for chart versions. Include a helm test hook (a Job with helm.sh/hook: test) that submits a minimal agent workload and asserts completion within 60 seconds, so operators can validate releases at any time.

Monitoring the Deployed Runtime

The Job Controller should expose three critical Prometheus metrics: agent_jobs_active (autoscaler input), agent_jobs_completed_total (success rate SLOs), and agent_job_duration_seconds (latency percentiles). Alert when active jobs exceed 90% of max_concurrent_jobs for 5+ minutes, success rate drops below 99%, or p99 duration exceeds the configured timeout.

Code Walkthrough

Now that you understand how ResourceQuota budgets, Redis durability, and DaemonSet log streaming fit together at the design level, the implementation centers on a single entry point: the Helm values file that drives every component in the agent-runtime release.

The two functions below represent the Job Controller's startup path. On pod launch, the controller reads a values file injected as a ConfigMap volume mount, validates that all required keys are present, then constructs a typed ControllerConfig shared across subsystems: the queue consumer reads queue_url, the RBAC binding uses service_account, and the scheduler enforces max_concurrent_jobs as its backpressure ceiling against the namespace ResourceQuota.

Code snippetpython
1import yaml
2from dataclasses import dataclass
3from pathlib import Path
4from typing import Optional
5
6@dataclass
7class ResourceProfile:
8    cpu_limit: str = "2"
9    memory_limit: str = "4Gi"
10    ephemeral_storage: str = "10Gi"
11
12@dataclass
13class ControllerConfig:
14    namespace: str
15    queue_url: str
16    max_concurrent_jobs: int
17    default_timeout_seconds: int
18    default_resources: ResourceProfile
19    service_account: str
20    image_pull_secret: Optional[str] = None
21    log_streamer_enabled: bool = True
22
23def load_helm_values(values_path: Path) -> dict:
24    required_keys = [("namespace",), ("queue", "url"), ("controller", "maxConcurrentJobs")]
25    with open(values_path, "r") as f:
26        values = yaml.safe_load(f)
27    for key_path in required_keys:
28        node = values
29        for part in key_path:
30            if not isinstance(node, dict) or part not in node:
31                raise KeyError(f"Missing required Helm value: {'.'.join(key_path)}")
32            node = node[part]
33    return values
34
35def build_controller_config(values: dict) -> ControllerConfig:
36    ctrl = values.get("controller", {})
37    res_raw = ctrl.get("resources", {})
38    resources = ResourceProfile(
39        cpu_limit=res_raw.get("cpuLimit", "2"),
40        memory_limit=res_raw.get("memoryLimit", "4Gi"),
41        ephemeral_storage=res_raw.get("ephemeralStorage", "10Gi"),
42    )
43    return ControllerConfig(
44        namespace=values["namespace"],
45        queue_url=values["queue"]["url"],
46        max_concurrent_jobs=ctrl["maxConcurrentJobs"],
47        default_timeout_seconds=ctrl.get("defaultTimeoutSeconds", 300),
48        default_resources=resources,
49        service_account=ctrl.get("serviceAccount", "agent-runtime-controller"),
50        image_pull_secret=values.get("imagePullSecret"),
51        log_streamer_enabled=values.get("logStreamer", {}).get("enabled", True),
52    )

A corresponding values.yaml snippet illustrates the expected shape. The controller.maxConcurrentJobs value of 20 pairs with a ResourceQuota capping the namespace at 40 CPU, leaving a 2-CPU-per-job budget with 20% headroom reserved for platform components. Setting logStreamer.enabled: true causes the DaemonSet to watch pods labeled agent-runtime.io/managed=true and forward their output to the configured external store.

Code snippetyaml
1namespace: agent-runtime
2imagePullSecret: registry-creds
3
4queue:
5  url: "redis://redis-master.agent-runtime.svc.cluster.local:6379/0"
6
7controller:
8  maxConcurrentJobs: 20
9  defaultTimeoutSeconds: 300
10  serviceAccount: agent-runtime-controller
11  resources:
12    cpuLimit: "2"
13    memoryLimit: "4Gi"
14    ephemeralStorage: "10Gi"
15
16logStreamer:
17  enabled: true

After deploying with helm upgrade --install agent-runtime ./helm/agent-runtime -f values.yaml, the Helm test hook — a Job annotated with helm.sh/hook: test — submits a minimal agent workload and asserts it completes within 60 seconds, giving operators a repeatable way to validate any release.

Verify by running helm test agent-runtime immediately after the upgrade succeeds and confirming the test pod exits with status Succeeded before promoting the release to production traffic.

Do's and Don'ts

Having walked through the controller's startup path and the Helm values that drive it, the following guardrails capture the configuration mistakes most likely to surface during a real helm upgrade --install of the agent runtime.

Do's

1. ✓Do set controller.maxConcurrentJobs to the namespace ResourceQuota CPU ceiling divided by cpuLimit — the Code Walkthrough's 20-job ceiling against a 40-CPU quota reserves a 20% buffer for platform components like the DaemonSet and Redis; breaching that ceiling causes Job pods to land in Pending indefinitely rather than failing fast.
2. ✓Do run helm test agent-runtime immediately after every helm upgrade --install — the test hook submits a real agent workload and asserts a 60-second completion, catching misconfigured queue.url, missing RBAC bindings, or a broken imagePullSecret before production traffic reaches the release.
3. ✓Do validate all three required Helm value paths (namespace, queue.url, controller.maxConcurrentJobs) at controller startup via load_helm_values — the validation runs before build_controller_config constructs the shared ControllerConfig, so a missing key raises a KeyError at boot rather than producing a ControllerConfig with a None queue URL that silently drops agent jobs at runtime.

Don'ts

1. ✗Don't omit logStreamer.enabled: true from values.yaml if you need agent output in production — the DaemonSet only watches pods labeled agent-runtime.io/managed=true when the flag is true; defaulting to False means agent job logs are never forwarded to the external store and failures become invisible without in-cluster log access.
2. ✗Don't set controller.maxConcurrentJobs independently of resources.cpuLimit in the Helm values — the backpressure ceiling only protects the namespace ResourceQuota when both values are sized together; raising maxConcurrentJobs without adjusting cpuLimit (or the quota) causes the namespace to exhaust CPU mid-burst and starve platform pods like the queue consumer itself.
3. ✗Don't skip the load_helm_values required-key check by passing a hand-constructed dict directly to build_controller_config — build_controller_config accesses ctrl["maxConcurrentJobs"] with a direct key lookup that raises KeyError at an unpredictable callsite; the structured path-traversal in load_helm_values is the only gate that catches partial configs before the controller begins scheduling jobs.