Deploy to Kubernetes with health check probes

Introduction

When you ship a FastAPI container to Kubernetes without health probes, the orchestrator routes traffic the instant the process starts — before models load, before connection pools warm — and the result is a burst of 502s every rollout. Liveness without a startup probe is worse: Kubernetes kills the pod mid-initialization, then kills it again, in a crash loop nobody asked for. By the end of this lesson you'll be able to write a Deployment manifest with all three probe types (startup, readiness, liveness) tuned for a GenAI API whose container takes 30-60 seconds to be ready and whose endpoints handle long-running inference requests.

Key Terminology

• Startup probe — a one-shot health check that runs first and blocks the other probes until it passes; protects slow-starting GenAI containers from being killed mid-warmup.
• Readiness probe — a continuous check that gates traffic by adding or removing the pod from the Service endpoint list; failure removes traffic without restarting the pod, which is the right response to a transient downstream outage.
• Liveness probe — a continuous check that restarts the container on failure; recovers from deadlocks and hung processes that a readiness probe can't fix.
• periodSeconds — how often, in seconds, a probe re-fires; combined with failureThreshold it sets the real-world window before the probe takes action, and it's tuned independently per probe type.
• failureThreshold — number of consecutive probe failures before action; multiplied by periodSeconds it sets the real-world window the pod has to recover.

Concepts

The three probe types and their lifecycle

Kubernetes provides three distinct probes that fire at different points in the container lifecycle. The startup probe runs first; while it's running, readiness and liveness are suspended. Once startup succeeds (or its failureThreshold × periodSeconds budget exhausts), it never runs again, and the other two take over for the rest of the container's life.

Why GenAI APIs need all three

GenAI services routinely take 30-60 seconds to be ready: model weights load from disk, embedding caches warm, connection pools to multiple LLM providers establish. Liveness alone would kill the container before initialization completed, looping forever. Readiness alone would route no traffic — but also never restart a hung pod. The startup probe protects the warmup window; readiness gates traffic during transient outages; liveness rescues stuck pods. All three are non-optional for production GenAI APIs (see Code Walkthrough).

Readiness, Services, and rolling-update cutover

The Service only routes traffic to pods that pass their readiness probe, so readiness is the lever that controls user-visible availability. A pod that fails readiness is removed from the Service endpoint list and receives no traffic — but it is not restarted, so a transient downstream hiccup costs you availability for that pod, not a cold start. During a rolling update this same gate decides when the cutover happens: the Deployment's RollingUpdate strategy (maxSurge: 1, maxUnavailable: 0) brings up a new pod and waits for its readiness probe to pass before any old pod is removed from rotation. Because the new pod only enters the Service endpoint list once it reports ready, traffic shifts only to warmed pods and the cluster never falls below the target replica count. Tuning the readiness probe's periodSeconds and failureThreshold therefore directly tunes how fast a fresh pod starts serving and how quickly a struggling one is pulled out.

Code Walkthrough

The concepts above — the probe lifecycle, why a GenAI API needs all three, and how readiness gates the rolling-update cutover — come together in a single Deployment manifest.

The Deployment manifest below combines all three concepts: a startup probe sized for a slow-loading GenAI container, a readiness probe tuned to take pods out of rotation on downstream failures, a liveness probe that restarts truly stuck containers, and a RollingUpdate strategy whose cutover is gated by the readiness probe.

Code snippetyaml
1# k8s/deployment.yaml
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: genai-api
6spec:
7  replicas: 3
8  selector:
9    matchLabels:
10      app: genai-api
11  strategy:
12    type: RollingUpdate
13    rollingUpdate:
14      maxSurge: 1
15      maxUnavailable: 0
16  template:
17    metadata:
18      labels:
19        app: genai-api
20    spec:
21      containers:
22        - name: genai-api
23          image: gcr.io/my-project/genai-api:latest
24          ports:
25            - containerPort: 8000
26              name: http
27          resources:
28            requests:
29              cpu: 250m
30              memory: 512Mi
31            limits:
32              cpu: "1"
33              memory: 1Gi
34          startupProbe:
35            httpGet:
36              path: /health
37              port: http
38            initialDelaySeconds: 5
39            periodSeconds: 5
40            failureThreshold: 12
41          readinessProbe:
42            httpGet:
43              path: /health
44              port: http
45            periodSeconds: 10
46            failureThreshold: 3
47            successThreshold: 1
48          livenessProbe:
49            httpGet:
50              path: /health
51              port: http
52            periodSeconds: 15
53            failureThreshold: 3
54            timeoutSeconds: 5

The startup probe (failureThreshold: 12, periodSeconds: 5) gives the container 60 seconds total — initialDelaySeconds plus failureThreshold × periodSeconds — to become healthy, which covers a typical model-load + connection-pool warmup. The readiness probe needs only 3 consecutive failures (30s) to take the pod out of rotation without restarting it: the right response to a temporary database or LLM-provider hiccup. The liveness probe uses a longer period (15s) and an explicit timeoutSeconds: 5 so a slow inference response doesn't trigger a phantom restart. Because maxUnavailable: 0 holds the old pod in the Service until the new pod's readiness probe passes, the rollout never routes traffic to a pod that is still warming up.

Verify by applying the manifest with kubectl apply -f k8s/deployment.yaml, watching kubectl get pods -w until all replicas reach READY 1/1, then running kubectl rollout restart deployment/genai-api. You'll know it works when the rollout completes with zero 502s at the Service endpoint and kubectl describe pod shows all three probes as passing.

Do's and Don'ts

Having tuned all three probes in the manifest above, keep them correct for a slow-starting, long-request GenAI API with the habits below — and watch for the traps that quietly break them.

Do's

1. ✓Do size the startup probe to exceed your worst observed cold-start time — measure it under load with model load + cache warm; don't guess at failureThreshold.
2. ✓Do keep /health cheap and side-effect-free — never have it call the database or downstream LLM provider, or a transient outage will cascade into pod restarts.
3. ✓Do tune the readiness probe's periodSeconds and failureThreshold for fast traffic cutover — a tighter period pulls a struggling pod out of the Service sooner and lets a fresh pod start serving promptly during a rolling update.

Don'ts

1. ✗Don't omit the startup probe on slow-starting containers — the liveness probe will kill the pod before it finishes loading, in an infinite restart loop.
2. ✗Don't use the same threshold for readiness and liveness — readiness should react fast (small threshold) to drain traffic; liveness should be slow (large threshold) so transient slowness never restarts a healthy pod.
3. ✗Don't point the liveness probe at the same heavy path you use for readiness — keep its timeoutSeconds generous and its period long so a slow inference response is never mistaken for a hung process and restarted.