Scale the chat API automatically with HPA based on CPU

Introduction

When your LLM chat API gets featured on Hacker News at 9am and traffic 10x's before you finish your coffee, manually running kubectl scale deployment llm-chat-api --replicas=10 is too slow — users hit timeouts before pods come up. Run it too early and you burn cash on idle pods. The Horizontal Pod Autoscaler (HPA) closes this loop: it watches metrics and adjusts the replica count automatically. By the end of this lesson you'll be able to define an HPA for a chat API Deployment, pick a sensible target utilization for variable inference workloads, and configure asymmetric scale-up / scale-down behavior so you absorb spikes fast without thrashing when traffic falls.

Key Terminology

• Horizontal Pod Autoscaler (HPA) — a Kubernetes controller that watches metrics and changes a Deployment's replica count to keep utilization near a target; this is the object you create to scale the chat API.
• Metrics Server — the cluster add-on that aggregates CPU/memory readings from each kubelet and exposes them via the Metrics API; HPA cannot make a decision without it.
• Target Utilization — the average resource usage HPA tries to maintain across pods (e.g. 60% CPU); the headroom between target and 100% is what absorbs LLM inference variance and pod-startup latency.
• Stabilization Window — a delay HPA enforces before acting on a sustained signal; the scale-down window in particular prevents flapping when traffic dips briefly.
• Scaling Behavior — the behavior block in autoscaling/v2 that sets independent stabilization windows and rate limits for scale-up vs scale-down, enabling fast-up / slow-down patterns for GenAI traffic.

Concepts

The HPA control loop

The HPA controller runs in the control plane on a loop. Every 15 seconds (configurable) it pulls metric values for the target Deployment's pods, then computes desiredReplicas = ceil(currentReplicas * (currentMetric / targetMetric)), clamps to minReplicas/maxReplicas, and updates the Deployment if the count changes. Example: 3 replicas averaging 90% CPU with a 60% target → ceil(3 * 90/60) = 5. The Deployment creates the new pods, the Service auto-discovers them via label selectors, load spreads, and average CPU drops back toward 60%.

Prerequisites HPA needs to function

HPA requires two things in the cluster. First, the Metrics Server must be installed and reporting — kubectl top pods -l app=llm-chat-api should return real numbers, not an error. Second, every pod in the target Deployment must declare resources.requests. HPA computes utilization as usage / request; without a request value there is no denominator and HPA refuses to scale (you'll see <unknown>/60% in kubectl get hpa). This is why resource-management work is an upstream dependency for autoscaling.

Choosing a target utilization for LLM inference

Set the target too high (90%) and a traffic burst pushes pods past 100% before new replicas can start — requests queue and tail latency spikes. Set it too low (30%) and you pay for pods that mostly idle. 60% is a strong default for chat APIs because per-request CPU varies wildly — a "Hello" prompt is cheap, a 32k-token chain-of-thought prompt is not — and the 40% headroom absorbs that variance; new pods also take time to clear the startup probe and join the Service, so existing pods carry the load during ramp; finally, the 15-second metrics interval means HPA reacts on a delay, and the headroom buys time.

Asymmetric scale-up and scale-down

Default scale-down has a 300-second stabilization window — HPA waits five minutes of sustained low utilization before removing pods, preventing flapping. For GenAI workloads the cost of under-provisioning (user-visible errors and timeouts) far exceeds the cost of running extra pods for a few extra minutes, so the production pattern is fast scale-up, slow scale-down: zero stabilization on the way up, a longer window on the way down. The behavior block in autoscaling/v2 is where you encode this asymmetry (see Code Walkthrough).

Code Walkthrough

Now that you've worked through the control loop, prerequisites, and the fast-up/slow-down rationale, the manifest below combines the two ideas that matter most in production: the scaleTargetRef + replica bounds + 60% CPU target from the basic HPA, and the asymmetric behavior block that makes scale-up aggressive and scale-down conservative.

1apiVersion: autoscaling/v2
2kind: HorizontalPodAutoscaler
3metadata:
4  name: llm-chat-api-hpa
5spec:
6  scaleTargetRef:
7    apiVersion: apps/v1
8    kind: Deployment
9    name: llm-chat-api
10  minReplicas: 2
11  maxReplicas: 10
12  metrics:
13  - type: Resource
14    resource:
15      name: cpu
16      target:
17        type: Utilization
18        averageUtilization: 60
19  behavior:
20    scaleUp:
21      stabilizationWindowSeconds: 0
22      policies:
23      - type: Percent
24        value: 100
25        periodSeconds: 15
26    scaleDown:
27      stabilizationWindowSeconds: 600
28      policies:
29      - type: Pods
30        value: 1
31        periodSeconds: 60

Apply it, then drive load to watch HPA react:

Code snippetbash
1kubectl apply -f hpa.yaml
2
3kubectl run load --image=busybox --restart=Never -- \
4  /bin/sh -c "while true; do wget -q -O- http://llm-chat-svc:8080/api/chat?prompt=hello; done"
5
6kubectl get hpa llm-chat-api-hpa --watch

You'll know it works when the TARGETS column shows utilization rising above 60%, the REPLICAS column climbs (e.g. 2 → 4 → 5) within a minute, and after you kubectl delete pod load, replicas stay at the elevated count for ~10 minutes before stepping down one pod at a time to the minReplicas floor of 2.

Do's and Don'ts

Having just assembled the manifest, use the following rules to keep that HPA healthy in production — what to do, and what to avoid.

Do's

1. ✓Do set resources.requests on every container in the target Deployment — HPA computes utilization as usage / request, and a missing request means no scaling.
2. ✓Do verify the Metrics Server is live before applying the HPA — kubectl top pods should return real numbers; without it the HPA sits in <unknown> and never scales.
3. ✓Do make scale-up faster than scale-down for chat APIs — user-visible timeouts cost more than a few minutes of extra pods.

Don'ts

1. ✗Don't set target utilization above 80% for inference workloads — per-request CPU variance plus pod-startup latency will push you past 100% before new replicas are Ready.
2. ✗Don't drop minReplicas to 1 — a floor of 1 lets a single bad pod take the API offline; keep it at 2 or higher for HA.
3. ✗Don't tune HPA before fixing slow pod startup — if pods take 90s to become Ready, no scaling policy will react fast enough; fix readiness/startup probes first.