Build K8s liveness/readiness probes with dependency monitoring

Introduction

When your API gateway pod loses its Redis connection mid-deployment, Kubernetes keeps routing traffic to it because the process is still alive — and every request bypasses your rate limiter until someone notices. Teams that conflate "process is running" with "pod can serve traffic" learn this distinction during incidents, not before. By the end of this lesson you'll be able to design Kubernetes-compatible liveness and readiness probe endpoints that distinguish process health from dependency health, so broken pods are pulled from rotation automatically instead of silently degrading the gateway.

Key Terminology

• Liveness probe: A Kubernetes mechanism that determines whether a pod's process is running and responsive; failure triggers a container restart.
• Readiness probe: A Kubernetes mechanism that determines whether a pod can accept traffic; failure removes the pod from service endpoints without restarting.
• Startup probe: A Kubernetes mechanism that gates liveness and readiness checks during initial boot, preventing premature restarts of slow-starting containers.

Concepts

Kubernetes Manifest Configuration

Configuring the probes correctly in your Kubernetes deployment manifest is just as important as the endpoint implementation. Misconfigured thresholds are a frequent source of cascading failures in production—an overly aggressive liveness probe can restart pods during normal garbage collection pauses, while an overly lenient readiness probe can leave broken pods in the load balancer for minutes.

Key configuration principles for an API gateway with the dependencies you have built:

• Liveness probe: Use a long initialDelaySeconds (at least 15 seconds) to account for application startup, SQLAlchemy connection pool initialization, and Redis connection establishment. Set failureThreshold to 3 with a periodSeconds of 10, meaning the pod must fail three consecutive checks over 30 seconds before Kubernetes restarts it. This prevents restarts from transient event loop stalls during garbage collection.

• Readiness probe: Use a shorter periodSeconds (5 seconds) because you want Kubernetes to quickly remove pods that lose Redis connectivity—every second a rate-limiter-less pod serves traffic is a second your abuse protection is bypassed. Set failureThreshold to 2 and successThreshold to 2, requiring two consecutive successes before the pod rejoins the service. The double success threshold prevents flapping when a dependency is intermittently available.

• Timeout discipline: Set timeoutSeconds to 3 for readiness probes. Your HealthCheckService runs checks concurrently, so the total latency is bounded by the slowest single dependency. If any dependency takes longer than 3 seconds to respond, the pod is functionally degraded for real-time gateway traffic and should not receive requests.

• Startup probes: For pods that take longer to initialize (pre-warming the content-addressable cache, loading PII scanning models for guardrails), use a startup probe with a generous failureThreshold of 30 and periodSeconds of 2, giving the pod up to 60 seconds to become ready before Kubernetes considers it a failed start. Startup probes disable liveness checks during initialization, preventing premature restarts of pods that are still loading.

Operational Patterns for Gateway Health Checks

Beyond the basic probe implementation, several operational patterns are essential for running health checks at scale across a fleet of API gateway pods:

• Tri-state dependency reporting: Treat each dependency check as a three-valued signal — HEALTHY, DEGRADED (functional but exhibiting elevated latency or resource pressure), and UNHEALTHY. The DEGRADED rung is an early-warning signal between green and red; readiness can still pass while alerting that a dependency is close to the edge.

• Cache warming and readiness: When a pod starts, its content-addressable response cache is empty. Depending on your traffic patterns, you may want the readiness probe to report DEGRADED during the first N seconds to signal to the load balancer that this pod will have higher latency than peers with warm caches. This is not a binary healthy/unhealthy decision—it is a nuanced signal that sophisticated load balancers (like Envoy with least-request routing) can use to gradually shift traffic to the new pod.

• Circuit breaker integration: Your readiness check for LLM providers should incorporate circuit breaker state. If the guardrail scanning circuit breaker has tripped (indicating the LLM provider has been unresponsive for several consecutive requests), the readiness check should report that dependency as UNHEALTHY without even attempting to reach the provider. This prevents the health check itself from contributing to connection pressure on an already overwhelmed downstream service.

• Structured logging for probe failures: Every transition from HEALTHY to DEGRADED or UNHEALTHY should emit a structured log event with the dependency name, the measured latency, the error message if any, and a correlation ID. During an incident, these log events provide a precise timeline of when each pod detected the dependency failure, which is essential for distinguishing between a dependency outage (all pods report simultaneously) and a network partition (only some pods report).

Taken together, these patterns turn the probe endpoints from simple HTTP checks into a precise signal channel between the pod and Kubernetes — one that distinguishes process failures from dependency failures, surfaces degraded conditions before they become outages, and gives operators the per-dependency timeline they need to diagnose incidents. That precision is what enables zero-downtime deployments, rolling updates, and automated incident recovery without manual intervention.

Code Walkthrough

Understanding Liveness vs. Readiness in a Gateway Context

Kubernetes exposes two distinct probe types that serve different purposes.

• Liveness probes answer: "Is this process deadlocked or corrupted?" A failed liveness probe triggers a pod restart. Liveness checks must be lightweight and must never depend on external services—if Redis is down, restarting the pod will not fix Redis.

• Readiness probes answer: "Can this pod handle traffic right now?" A failed readiness probe removes the pod from the Service's endpoint list. For your gateway, readiness failure means the pod cannot enforce rate limits or run guardrail scans.

If your Redis connection drops, the correct response is to mark the pod as not-ready (diverting traffic) rather than killing it (which wastes time on container restarts).

Loading diagram...

This diagram captures the fundamental architectural split: liveness inspects only the process itself, while readiness fans out to every dependency that the gateway's middleware stack requires. The database supports audit trail persistence for guardrail events, Redis backs both the rate limiter and the response cache, and LLM providers power the PII scanning guardrails.

Implementing the HealthCheckService

The core of a robust health check system is a service class that encapsulates individual dependency checks, runs them concurrently to minimize probe latency, and aggregates results into a structured response. The following implementation defines a HealthCheckService class with three key methods: check_database executes a SELECT 1 query through async SQLAlchemy and measures round-trip latency, check_redis calls the Redis PING command and reports connection pool statistics that reveal pool exhaustion before it causes failures, and check_llm_providers sends a minimal completion request to verify the guardrail scanning pipeline is functional. All three checks run concurrently via asyncio.gather to keep the total probe response time bounded.

1import asyncio
2import time
3from dataclasses import dataclass, field
4from enum import Enum
5from typing import Optional
6
7import httpx
8import redis.asyncio as aioredis
9from sqlalchemy import text
10from sqlalchemy.ext.asyncio import AsyncSession
11
12class DependencyStatus(Enum):
13    HEALTHY = "healthy"
14    DEGRADED = "degraded"
15    UNHEALTHY = "unhealthy"
16
17@dataclass
18class CheckResult:
19    name: str
20    status: DependencyStatus
21    latency_ms: float
22    message: str = ""
23    metadata: dict = field(default_factory=dict)
24
25class HealthCheckService:
26    LATENCY_WARN_MS = 500
27    LATENCY_FAIL_MS = 2000
28
29    def __init__(self, db_session_factory, redis_client: aioredis.Redis,
30                 llm_base_url: str, llm_api_key: str):
31        self._db_factory = db_session_factory
32        self._redis = redis_client
33        self._llm_url = llm_base_url
34        self._llm_key = llm_api_key
35
36    async def check_database(self) -> CheckResult:
37        start = time.monotonic()
38        try:
39            async with self._db_factory() as session:
40                await session.execute(text("SELECT 1"))
41            latency = (time.monotonic() - start) * 1000
42            status = (DependencyStatus.DEGRADED
43                      if latency > self.LATENCY_WARN_MS
44                      else DependencyStatus.HEALTHY)
45            return CheckResult("database", status, latency)
46        except Exception as exc:
47            latency = (time.monotonic() - start) * 1000
48            return CheckResult("database", DependencyStatus.UNHEALTHY,
49                               latency, message=str(exc))
50
51    async def check_redis(self) -> CheckResult:
52        start = time.monotonic()
53        try:
54            await self._redis.ping()
55            latency = (time.monotonic() - start) * 1000
56            pool = self._redis.connection_pool
57            pool_info = {
58                "active_connections": len(pool._in_use_connections),
59                "available_connections": len(pool._available_connections),
60                "max_connections": pool.max_connections,
61            }
62            utilization = len(pool._in_use_connections) / pool.max_connections
63            status = (DependencyStatus.DEGRADED if utilization > 0.8
64                      else DependencyStatus.HEALTHY)
65            return CheckResult("redis", status, latency, metadata=pool_info)
66        except Exception as exc:
67            latency = (time.monotonic() - start) * 1000
68            return CheckResult("redis", DependencyStatus.UNHEALTHY,
69                               latency, message=str(exc))
70
71    async def check_llm_providers(self) -> CheckResult:
72        start = time.monotonic()
73        try:
74            async with httpx.AsyncClient(timeout=2.0) as client:
75                resp = await client.post(
76                    f"{self._llm_url}/v1/chat/completions",
77                    headers={"Authorization": f"Bearer {self._llm_key}"},
78                    json={"model": "ping", "messages": [{"role": "user", "content": "."}],
79                          "max_tokens": 1},
80                )
81            latency = (time.monotonic() - start) * 1000
82            if resp.status_code >= 500:
83                return CheckResult("llm_providers", DependencyStatus.UNHEALTHY,
84                                   latency, message=f"HTTP {resp.status_code}")
85            status = (DependencyStatus.DEGRADED
86                      if latency > self.LATENCY_WARN_MS
87                      else DependencyStatus.HEALTHY)
88            return CheckResult("llm_providers", status, latency)
89        except Exception as exc:
90            latency = (time.monotonic() - start) * 1000
91            return CheckResult("llm_providers", DependencyStatus.UNHEALTHY,
92                               latency, message=str(exc))
93
94    async def check_all(self) -> dict:
95        results = await asyncio.gather(
96            self.check_database(),
97            self.check_redis(),
98            self.check_llm_providers(),
99            return_exceptions=True,
100        )
101        checks = []
102        for r in results:
103            if isinstance(r, Exception):
104                checks.append(CheckResult("unknown", DependencyStatus.UNHEALTHY,
105                                          0, message=str(r)))
106            else:
107                checks.append(r)
108        overall = DependencyStatus.HEALTHY
109        for c in checks:
110            if c.status == DependencyStatus.UNHEALTHY:
111                overall = DependencyStatus.UNHEALTHY
112                break
113            if c.status == DependencyStatus.DEGRADED:
114                overall = DependencyStatus.DEGRADED
115        return {"status": overall.value, "checks": [
116            {"name": c.name, "status": c.status.value,
117             "latency_ms": round(c.latency_ms, 2),
118             "message": c.message, **c.metadata} for c in checks
119        ]}

Exposing Probe Endpoints with FastAPI

With the health check service built, you need HTTP endpoints that Kubernetes can poll. The following code defines a FastAPI router with two endpoints: /healthz for liveness and /readyz for readiness. The liveness endpoint performs only a trivial in-process check and returns True immediately, while the readiness endpoint delegates to the HealthCheckService.check_all method and returns an appropriate HTTP status code. Notice how the readiness endpoint returns HTTP 503 (Service Unavailable) when any dependency is unhealthy, which Kubernetes interprets as a readiness failure without ambiguity.

1from fastapi import APIRouter, Response, Depends
2from starlette.status import HTTP_200_OK, HTTP_503_SERVICE_UNAVAILABLE
3
4router = APIRouter(tags=["health"])
5
6async def get_health_service() -> HealthCheckService:
7    # Wired via dependency injection in your app factory
8    ...
9
10@router.get("/healthz")
11async def liveness():
12    return {"status": "alive"}
13
14@router.get("/readyz")
15async def readiness(
16    response: Response,
17    svc: HealthCheckService = Depends(get_health_service),
18):
19    result = await svc.check_all()
20    if result["status"] == "unhealthy":
21        response.status_code = HTTP_503_SERVICE_UNAVAILABLE
22    return result
23
24@router.get("/readyz/{dependency}")
25async def readiness_single(
26    dependency: str,
27    response: Response,
28    svc: HealthCheckService = Depends(get_health_service),
29):
30    check_map = {
31        "database": svc.check_database,
32        "redis": svc.check_redis,
33    }
34    check_fn = check_map.get(dependency)
35    if check_fn is None:
36        response.status_code = 404
37        return {"error": f"Unknown dependency: {dependency}"}
38    result = await check_fn()
39    if result.status == DependencyStatus.UNHEALTHY:
40        response.status_code = HTTP_503_SERVICE_UNAVAILABLE
41    return {
42        "name": result.name, "status": result.status.value,
43        "latency_ms": round(result.latency_ms, 2),
44        "message": result.message, **result.metadata,
45    }

Do's and Don'ts

Do's

1. ✓Do keep /healthz liveness checks strictly process-internal — only inspect the event loop and memory state inside the liveness endpoint; if Redis or the database is down, restarting the pod solves nothing, and a liveness probe that calls check_redis converts a dependency outage into a cascading pod-restart loop that prolongs the incident.
2. ✓Do run all three HealthCheckService dependency checks concurrently via asyncio.gather — check_database, check_redis, and check_llm_providers each block on network I/O, so serializing them multiplies probe latency and risks blowing past Kubernetes' timeoutSeconds budget, which marks a healthy pod as not-ready for no real reason.
3. ✓Do surface DependencyStatus.DEGRADED as a distinct intermediate state by checking Redis pool utilization and comparing latency_ms against LATENCY_WARN_MS = 500 — a pool at 90% utilization (_in_use_connections / max_connections > 0.8) or a database rounding 1,900 ms is not yet failing but is about to; the DEGRADED signal gives operators time to act before the pod is pulled from the Service endpoint list.

Don'ts

1. ✗Don't call check_redis, check_database, or check_llm_providers from the /healthz liveness handler — liveness failure triggers a pod restart, so wiring an external dependency into /healthz means an unavailable Redis instance kills every running pod in the deployment simultaneously, turning a recoverable dependency blip into a full service outage.
2. ✗Don't collapse CheckResult.status to a binary healthy/unhealthy without the DEGRADED tier — skipping the LATENCY_WARN_MS branch means a Redis pool at 95% capacity or a SELECT 1 returning in 1,900 ms still reports HEALTHY, hiding the signal entirely until the check tips into UNHEALTHY and traffic is already failing.
3. ✗Don't treat a not-ready readiness outcome and a failed liveness outcome as interchangeable actions — a dropped Redis connection should remove the pod from the Service's endpoint list (readiness failure, traffic diverted, pod kept alive), not restart it; conflating the two wastes container start time, breaks in-flight rate-limit state, and does nothing to restore the Redis connection.