Configure GKE deployments with HPA on custom metrics

Introduction

When you deploy a GenAI application to GKE with default autoscaling, your first real traffic spike will quietly degrade user experience while Kubernetes reports everything healthy — pods sit at 35% CPU because inference is I/O-bound on upstream model calls, the HPA does nothing, and p99 latency climbs from 2 seconds to 45 seconds as connection pools exhaust. The fix is to drive autoscaling on the signals that actually correlate with LLM saturation: in-flight request count and queue depth. By the end of this lesson you'll be able to configure a GKE Deployment with a Horizontal Pod Autoscaler driven by custom genai_inflight_requests metrics, a PodDisruptionBudget that protects availability during node drains, and a zero-downtime rolling update strategy.

Key Terminology

• Horizontal Pod Autoscaler (HPA): Kubernetes controller that adjusts a Deployment's replica count based on observed metric values; for GenAI workloads we drive it from custom metrics rather than CPU.
• Custom metric (genai_inflight_requests): Per-pod Prometheus gauge tracking concurrent in-flight LLM requests, exposed by the FastAPI middleware and consumed by the HPA via prometheus-adapter.
• prometheus-adapter: Cluster component that translates Prometheus series into the Kubernetes custom.metrics.k8s.io API so the HPA can read application-defined signals.
• PodDisruptionBudget (PDB): Policy object enforcing a minAvailable count of pods during voluntary disruptions (node drain, upgrade, autoscaler eviction).
• Rolling update strategy (maxSurge / maxUnavailable): Deployment field controlling how many extra pods may be created and how many may be unavailable during a rollout; maxSurge: 1 / maxUnavailable: 0 is the zero-downtime configuration used here.

Concepts

The pipeline below shows how the three custom metrics defined in the FastAPI middleware reach the HPA controller and feed scaling decisions on the GenAI Deployment, while the PDB constrains how many replicas the scheduler may take offline during voluntary disruptions.

The HPA reads the average value of genai_inflight_requests across pods and compares it to the target (30 per pod). When the actual value exceeds the target, the controller increases replicas, bounded by maxReplicas and the behavior.scaleDown stabilization window. The PDB does not influence scaling but it blocks node-drain operations that would push the replica count below minAvailable, which preserves availability during cluster upgrades and spot preemptions.

Code Walkthrough

Now that you've seen how the full pipeline connects — FastAPI middleware publishing genai_inflight_requests to Prometheus, the prometheus-adapter translating that series into custom.metrics.k8s.io, and the HPA controller reading the per-pod average to drive replica counts — the two artifacts below implement each end of that pipeline.

Exposing the custom metrics from FastAPI

The GenAIMetricsMiddleware class increments the genai_inflight_requests gauge on every request entry and decrements it on exit, producing the live concurrency signal the HPA targets. genai_request_queue_depth is updated by your connection-pool manager wherever requests wait for an available LLM slot. Both gauges, plus the genai_tokens_total counter, map exactly to the three metric names the prometheus-adapter is configured to scrape.

Code snippetpython
1# genai_metrics.py
2import time
3from prometheus_client import Gauge, Counter, generate_latest
4from starlette.middleware.base import BaseHTTPMiddleware
5from starlette.requests import Request
6from starlette.responses import Response
7from fastapi import FastAPI
8
9INFLIGHT_REQUESTS = Gauge(
10    "genai_inflight_requests",
11    "Concurrent in-flight LLM requests per pod",
12    ["model_backend"],
13)
14QUEUE_DEPTH = Gauge(
15    "genai_request_queue_depth",
16    "Requests waiting for an available LLM connection",
17)
18TOKENS_TOTAL = Counter(
19    "genai_tokens_total",
20    "Total tokens generated",
21    ["direction"],
22)
23
24class GenAIMetricsMiddleware(BaseHTTPMiddleware):
25    async def dispatch(self, request: Request, call_next):
26        backend = request.headers.get("x-model-backend", "default")
27        INFLIGHT_REQUESTS.labels(model_backend=backend).inc()
28        try:
29            return await call_next(request)
30        finally:
31            INFLIGHT_REQUESTS.labels(model_backend=backend).dec()
32
33def create_app() -> FastAPI:
34    app = FastAPI(title="GenAI Service")
35    app.add_middleware(GenAIMetricsMiddleware)
36    app.add_route(
37        "/metrics",
38        lambda _: Response(generate_latest(), media_type="text/plain; version=0.0.4"),
39        methods=["GET"],
40    )
41    return app

Wiring the HPA and PodDisruptionBudget

With the metric exposed at /metrics, the HPA manifest tells the controller to maintain an average of 30 in-flight requests per pod, scaling between 2 and 10 replicas. The scaleDown.stabilizationWindowSeconds: 120 prevents the controller from shedding replicas immediately after a burst subsides. The PodDisruptionBudget sits alongside the Deployment and blocks node-drain operations that would push the live replica count below 2, which is the protection point during cluster upgrades and spot preemptions described in the Concepts diagram. The Deployment's rollingUpdate fields — maxSurge: 1 and maxUnavailable: 0 — ensure new pods become ready before old ones are terminated, so serving capacity never drops during a rollout.

Code snippetyaml
1# hpa-and-pdb.yaml
2apiVersion: autoscaling/v2
3kind: HorizontalPodAutoscaler
4metadata:
5  name: genai-hpa
6spec:
7  scaleTargetRef:
8    apiVersion: apps/v1
9    kind: Deployment
10    name: genai-deployment
11  minReplicas: 2
12  maxReplicas: 10
13  metrics:
14    - type: Pods
15      pods:
16        metric:
17          name: genai_inflight_requests
18        target:
19          type: AverageValue
20          averageValue: "30"
21  behavior:
22    scaleDown:
23      stabilizationWindowSeconds: 120
24---
25apiVersion: policy/v1
26kind: PodDisruptionBudget
27metadata:
28  name: genai-pdb
29spec:
30  minAvailable: 2
31  selector:
32    matchLabels:
33      app: genai-deployment

Verify by running kubectl describe hpa genai-hpa and confirming the AverageValue target of 30 appears alongside a live current metric reading sourced from custom.metrics.k8s.io.

Do's and Don'ts

Do's

1. ✓Do drive your HPA target metric from genai_inflight_requests rather than CPU — LLM inference is I/O-bound on upstream model calls, so CPU sits at 35% even while p99 latency climbs to 45 seconds; only concurrency-based metrics reflect actual saturation.
2. ✓Do set scaleDown.stabilizationWindowSeconds: 120 on the HPA — without it the controller sheds replicas immediately after a burst subsides, causing oscillation that re-triggers scale-up events before new pods are ready to serve.
3. ✓Do pair the Deployment's maxUnavailable: 0 / maxSurge: 1 rolling-update settings with a PodDisruptionBudget of minAvailable: 2 — the rolling-update fields ensure capacity never drops during a rollout, while the PDB independently blocks node-drain operations (cluster upgrades, spot preemptions) from pushing the live replica count below the safe floor.

Don'ts

1. ✗Don't omit the finally block in GenAIMetricsMiddleware.dispatch — if an exception escapes call_next before the decrement fires, genai_inflight_requests permanently overcounts, causing the HPA to scale replicas based on a gauge that only ever increases and never reflects real concurrency.
2. ✗Don't configure the HPA to read from Prometheus directly — the controller can only consume metrics exposed through custom.metrics.k8s.io; skipping the prometheus-adapter translation layer means kubectl describe hpa genai-hpa will report unknown for the current metric value and autoscaling will never trigger.
3. ✗Don't set minReplicas: 1 when using a PodDisruptionBudget with minAvailable: 2 — the PDB's floor exceeds the HPA's minimum, so any node-drain attempt will be permanently blocked because Kubernetes cannot legally evict a single pod without violating the disruption budget.