Build agent runtime auto-scaling and queue depth metrics

Introduction

In production, agent runtimes face a fundamental tension: provision too many executor pods and you waste compute budget on idle containers; provision too few and jobs queue indefinitely, violating latency SLAs. Unlike web services that scale on CPU or memory, agent workloads require queue-depth-aware scaling — the decision to add or remove executor capacity must derive from pending job counts, weighted by priority tier. This lesson teaches you how to implement an AgentQueueManager that enforces backpressure, exports queue-depth metrics to Prometheus, and drives autoscaling decisions using a priority-weighted formula so your executor pool expands aggressively for critical interactive sessions without over-provisioning for batch workloads.

Key Terminology

• Queue Depth: The total number of pending (unstarted) agent jobs across all priority tiers at a given instant; the primary signal for scaling decisions.
• Scaling Cooldown: A configurable time window after a scale-up or scale-down event during which additional scaling actions are suppressed, preventing oscillation.
• Backpressure: A mechanism that rejects or delays new job submissions when the queue depth exceeds a configured ceiling, protecting executor pods and downstream sandboxed environments from overload.
• Drain Time: The estimated duration to process all pending jobs given the current executor count and average job duration; used to project whether SLA targets will be met.
• Executor Pool: The set of Kubernetes pods available to pick up and run agent jobs, each provisioned with resource isolation (CPU/memory limits, network policies) as defined by your sandboxing layer.

Concepts

Scaling Strategy: Priority-Weighted Queue Depth

The priority-weighted approach used in compute_scaling_metrics deserves deeper examination. A naive scaling strategy that treats all jobs equally creates a dangerous failure mode: if 200 batch jobs are enqueued simultaneously, the autoscaler spins up dozens of executor pods, consuming cluster capacity that should remain available for critical interactive sessions. By assigning weights — 3× for critical, 1× for standard, 0.5× for batch — a single critical job exerts the same scaling pressure as six batch jobs. This means the system scales aggressively for interactive workloads while treating batch backlogs as lower-urgency signals.

The jobs_per_executor parameter controls concurrency within each pod. Setting this to 2 means each executor pod runs two agent jobs simultaneously within separate sandboxed containers. This value must align with the resource limits defined in your pod spec — if each agent job requests 1 CPU and 2 GiB memory, and your executor pods have 4 CPU / 8 GiB limits, then jobs_per_executor of 2 leaves headroom for the executor's own overhead. Setting it too high causes memory pressure and OOM kills inside the sandbox; setting it to 1 wastes resources on pod overhead per job.

Backpressure and Admission Control

The enqueue method's backpressure mechanism — returning False when total queue depth hits max_queue_depth — is intentionally simple. In production, you should layer additional admission controls:

1. Rate limiting per tenant — prevent a single user from flooding the queue. Implement a per-tenant token bucket that gates calls to enqueue, returning HTTP 429 with a Retry-After header when the bucket is empty.
2. Priority-aware admission — when the queue is 80% full, reject only batch-priority submissions. This preserves capacity for critical and standard jobs during load spikes, degrading gracefully rather than cutting off all submissions.
3. Estimated wait time — use estimated_drain_seconds from ScalingMetrics to return an X-Estimated-Wait header on successful enqueue responses. Clients can use this to set appropriate timeouts or display progress indicators.

Connecting Scaling to Execution Monitoring

The autoscaler does not operate in isolation — it must feed data back into your execution monitoring pipeline. Every scale event should emit a structured log entry and a Prometheus annotation so that when you investigate job latency spikes on your dashboards, you can correlate them with scaling transitions. The _apply_scale method's log statement is a starting point, but production deployments should also write Kubernetes Events on the deployment object (using the CoreV1Api.create_namespaced_event method) so that kubectl describe deployment agent-executor surfaces scaling history directly.

Additionally, connect your mark_complete callback to update active-job gauges, allowing the scaling decision engine to incorporate not just pending depth but also in-flight concurrency. If 40 jobs are pending and 50 are actively running across 25 executors, the system is at full capacity and should scale up — even though the queue depth alone might suggest moderate load. The combination of pending depth and active concurrency gives the most accurate picture of required executor capacity.

Code Walkthrough

Now that you understand the priority-weighted scaling strategy and backpressure mechanics from the Concepts section, the code below assembles those ideas into a working AgentQueueManager.

The class maintains three deque instances — one per priority tier — and exposes three methods that put the architectural principles into practice. The enqueue method checks total queue depth against max_queue_depth before accepting a job; when the ceiling is reached it returns False, signaling the submission layer to apply backpressure rather than silently dropping work or overloading executor pods. The compute_scaling_metrics method applies the priority weights (3× for critical, 1× for standard, 0.5× for batch) to produce a weighted depth, divides by jobs_per_executor to calculate how many executor pods the backlog justifies, then clamps the result between min_replicas and max_replicas. Both Prometheus gauges — queue depth per tier and desired replica count — update on every call, giving your monitoring stack a continuous view of queue state without a separate polling component.

Code snippetpython
1import time
2from collections import deque
3from dataclasses import dataclass, field
4from enum import Enum
5
6from prometheus_client import Gauge, Histogram
7
8class Priority(Enum):
9    CRITICAL = "critical"
10    STANDARD = "standard"
11    BATCH = "batch"
12
13QUEUE_DEPTH = Gauge("agent_queue_depth", "Pending agent jobs", ["priority"])
14SCALING_DESIRED = Gauge("agent_scaling_desired_replicas", "Target executor replica count")
15DRAIN_TIME = Histogram("agent_queue_drain_time_seconds", "Estimated queue drain time")
16
17@dataclass
18class ScalingConfig:
19    jobs_per_executor: int = 2
20    min_replicas: int = 1
21    max_replicas: int = 50
22    max_queue_depth: int = 500
23    avg_job_duration_seconds: float = 120.0
24    cooldown_seconds: float = 30.0
25    priority_weights: dict = field(
26        default_factory=lambda: {"critical": 3, "standard": 1, "batch": 0.5}
27    )
28
29@dataclass
30class ScalingMetrics:
31    total_depth: int = 0
32    depth_by_priority: dict = field(default_factory=dict)
33    desired_replicas: int = 0
34    estimated_drain_seconds: float = 0.0
35    last_computed: float = field(default_factory=time.time)
36
37class AgentQueueManager:
38    def __init__(self, config: ScalingConfig):
39        self._config = config
40        self._queues = {p: deque() for p in Priority}
41        self._last_scale_time = 0.0
42
43    def enqueue(self, job_id: str, priority: Priority) -> bool:
44        total = sum(len(q) for q in self._queues.values())
45        if total >= self._config.max_queue_depth:
46            return False  # backpressure: reject submissions above ceiling
47        self._queues[priority].append(job_id)
48        QUEUE_DEPTH.labels(priority=priority.value).inc()
49        return True
50
51    def dequeue(self, priority: Priority) -> "str | None":
52        if self._queues[priority]:
53            job_id = self._queues[priority].popleft()
54            QUEUE_DEPTH.labels(priority=priority.value).dec()
55            return job_id
56        return None
57
58    def compute_scaling_metrics(self) -> ScalingMetrics:
59        weights = self._config.priority_weights
60        depth_by_priority = {p.value: len(self._queues[p]) for p in Priority}
61        weighted_depth = sum(
62            depth_by_priority[p.value] * weights.get(p.value, 1)
63            for p in Priority
64        )
65        total_depth = sum(depth_by_priority.values())
66        desired = max(
67            self._config.min_replicas,
68            min(
69                self._config.max_replicas,
70                int(weighted_depth / self._config.jobs_per_executor) + 1,
71            ),
72        )
73        drain_seconds = (
74            (total_depth / (desired * self._config.jobs_per_executor))
75            * self._config.avg_job_duration_seconds
76            if desired > 0
77            else 0.0
78        )
79        SCALING_DESIRED.set(desired)
80        DRAIN_TIME.observe(drain_seconds)
81        return ScalingMetrics(
82            total_depth=total_depth,
83            depth_by_priority=depth_by_priority,
84            desired_replicas=desired,
85            estimated_drain_seconds=drain_seconds,
86        )

The ScalingMetrics snapshot returned by compute_scaling_metrics is what your autoscaler loop consumes on each tick. Feed desired_replicas into a Kubernetes Scale API call, and gate consecutive scale events behind the cooldown_seconds window — tracked via _last_scale_time — to prevent the oscillation that naive reactive scaling produces when a burst of batch jobs temporarily inflates weighted depth and then drains quickly. The estimated_drain_seconds field lets you detect SLA risk before it materializes: if drain time exceeds your latency target, alert before users notice.

Check that your implementation works correctly by constructing an AgentQueueManager with max_queue_depth=5, enqueuing five jobs at Priority.STANDARD, then attempting a sixth — the sixth call must return False, confirming that backpressure engages precisely at the configured ceiling.

Do's and Don'ts

Do's

1. ✓Do scale on queue depth, not CPU utilization — Agent jobs have irregular CPU profiles (idle during LLM API calls, bursting during tool execution). CPU-based HPA reacts too slowly and incorrectly to these patterns. Queue depth directly measures demand.
2. ✓Do enforce cooldown windows between scaling events — Without cooldown, the autoscaler oscillates: it scales up, new pods start draining the queue, depth drops, it scales down, depth rises again. A 30-60 second cooldown lets new pods absorb load before the next decision.
3. ✓Do weight priorities differently in the scaling formula — Treating all jobs equally causes batch floods to starve interactive sessions. Priority weighting ensures resource allocation aligns with business criticality.

Don'ts

1. ✗Don't set max_replicas without considering cluster capacity — An uncapped max_replicas can exhaust node pool resources, causing pending pods across all namespaces. Calculate your maximum based on node count × allocatable resources ÷ per-executor resource requests.
2. ✗Don't skip drain time estimation in your scaling metrics — Drain time is the SLA-facing metric. A queue depth of 100 means nothing without knowing whether it drains in 2 minutes or 2 hours. Always compute and expose it alongside depth.
3. ✗Don't use a single queue without priority tiers — A FIFO queue where batch and critical jobs compete causes head-of-line blocking. A 10-minute batch job submitted before a critical interactive session delays the critical job unacceptably.