Instrument endpoints with Prometheus metrics

Introduction

When you deploy a GenAI API to Kubernetes, container metrics from the kubelet show CPU and memory but reveal nothing about whether requests are succeeding, how long the LLM provider is taking, or which model is burning tokens. Teams that rely on container-level signals alone discover regressions only after users complain — a 5xx spike on /v1/completions or a 10× jump in output-token consumption stays invisible until the bill arrives or the support queue fills. By the end of this lesson you will be able to instrument a FastAPI application with both automatic HTTP metrics and custom GenAI counters, histograms, and gauges, expose them on /metrics for Prometheus to scrape, and query them with PromQL.

Key Terminology

• Counter — a monotonically increasing metric that resets only on process restart; the right shape for total request counts, total errors, and cumulative tokens consumed.
• Histogram — observations bucketed at fixed boundaries so Prometheus can compute percentiles (p50/p95/p99) server-side; the right shape for request duration and response size.
• Gauge — a point-in-time value that can go up or down; the right shape for active streaming connections, queue depth, and current memory usage.
• Instrumentator — the prometheus-fastapi-instrumentator class that auto-captures per-route HTTP metrics and exposes the /metrics endpoint with one initialization call.
• PromQL — Prometheus's query language; rate() over counter metrics and histogram_quantile() over histogram buckets are the two expressions you will reach for daily.

Concepts

Choosing the right metric type

Picking the wrong metric type makes a measurement difficult or impossible to query later. The decision tree below captures the four-way choice. Counters and histograms cover most GenAI use cases, gauges handle in-flight state, and summaries are usually wrong in Kubernetes because they pre-compute quantiles per pod and cannot be aggregated across replicas.

Automatic HTTP metrics vs custom GenAI metrics

prometheus-fastapi-instrumentator registers ASGI middleware that captures request count, request duration, and response size for every route — one instrument(app) call covers the entire HTTP surface. What it cannot capture is anything the framework does not see: tokens consumed per LLM call, which model handled the request, whether a prompt-template cache hit, or how many SSE streams are currently open. Those signals require custom counters, histograms, and gauges defined directly against prometheus_client and incremented from your handler code (see Code Walkthrough).

Latency buckets for LLM-backed endpoints

The default histogram buckets in most instrumentation libraries top out around 1 second, which is fine for typical CRUD APIs but loses all signal for LLM-backed routes that routinely take 2–10 seconds and stretch to 30 seconds for long-context completions. Without extended buckets every slow request lands in the +Inf bucket and histogram_quantile cannot tell you whether p99 is 5 s or 25 s. Define buckets that span 10 ms to 30 s and reuse the same bucket boundaries for both HTTP duration and LLM-provider duration so the two are directly comparable.

Querying with PromQL

rate(http_requests_total[5m]) gives per-second request rate over a 5-minute window; add label filters like {handler="/v1/completions", status="200"} to slice. histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) returns p99 latency. rate(llm_tokens_total{token_type="output"}[1h]) * 3600 gives hourly output-token volume — multiply by the per-token price for cost. rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) is your error-rate SLI; alert when it crosses 1%.

Code Walkthrough

The first snippet ties together three concepts from the previous section: configuring the instrumentator with LLM-friendly buckets, declaring custom GenAI counters and histograms against prometheus_client, and wiring everything into FastAPI's lifespan so /metrics is live before the first request is served.

Code snippetpython
1# app/metrics.py
2from contextlib import asynccontextmanager
3from fastapi import FastAPI
4from prometheus_fastapi_instrumentator import Instrumentator
5from prometheus_client import Counter, Histogram, Gauge
6
7DURATION_BUCKETS = (
8    0.01, 0.025, 0.05, 0.1, 0.25, 0.5,
9    1.0, 2.5, 5.0, 10.0, 15.0, 30.0,
10)
11
12instrumentator = Instrumentator(
13    should_group_status_codes=False,
14    should_ignore_untemplated=True,
15    excluded_handlers=["/health", "/metrics"],
16    inprogress_name="http_requests_inprogress",
17    inprogress_labels=True,
18)
19
20llm_tokens_total = Counter(
21    "llm_tokens_total",
22    "Total LLM tokens consumed",
23    labelnames=["model", "token_type", "endpoint"],
24)
25
26llm_request_duration = Histogram(
27    "llm_request_duration_seconds",
28    "Time spent waiting for LLM provider response",
29    labelnames=["model", "provider"],
30    buckets=DURATION_BUCKETS,
31)
32
33active_streams = Gauge(
34    "active_streams",
35    "Currently open SSE streaming connections",
36    labelnames=["model"],
37)
38
39@asynccontextmanager
40async def lifespan(app: FastAPI):
41    instrumentator.instrument(app)
42    instrumentator.expose(app, include_in_schema=False, should_gzip=True)
43    yield
44
45app = FastAPI(title="GenAI API", lifespan=lifespan)

• DURATION_BUCKETS — explicit boundaries from 10 ms to 30 s so LLM tail latency is visible; reused for both HTTP and LLM histograms.
• Instrumentator(...) — should_group_status_codes=False keeps 401 and 403 distinct; excluding /health and /metrics prevents internal traffic from skewing rates.
• llm_tokens_total — three labels (model, token_type, endpoint) let you query "output tokens for gpt-4o on /v1/completions" with a single PromQL expression.
• active_streams — a Gauge because streams open and close; counters cannot represent in-flight state.
• lifespan(...) — instrument() registers the middleware, expose() mounts /metrics; both must run before traffic arrives, which is exactly what FastAPI's lifespan guarantees.

The handler then increments the custom metrics around the LLM call:

Code snippetpython
1# app/routes/completions.py
2import time
3from fastapi import APIRouter
4from app.metrics import llm_tokens_total, llm_request_duration
5from app.llm import get_completion
6
7router = APIRouter(prefix="/v1", tags=["completions"])
8
9@router.post("/completions")
10async def create_completion(body: CompletionRequest):
11    model = body.model
12    start = time.monotonic()
13    result = await get_completion(
14        model=model, messages=body.messages, max_tokens=body.max_tokens
15    )
16    duration = time.monotonic() - start
17
18    llm_request_duration.labels(model=model, provider="openai").observe(duration)
19    llm_tokens_total.labels(
20        model=model, token_type="input", endpoint="/v1/completions"
21    ).inc(result.usage.prompt_tokens)
22    llm_tokens_total.labels(
23        model=model, token_type="output", endpoint="/v1/completions"
24    ).inc(result.usage.completion_tokens)
25
26    return {"id": result.id, "content": result.choices[0].message.content}

time.monotonic() is immune to wall-clock adjustments, so the duration recorded in the histogram is always non-negative. inc(n) adds n in a single call, which is how you record batch token counts from one LLM response.

You'll know it works when curl http://localhost:8000/metrics returns a text response containing http_requests_total, llm_tokens_total, and llm_request_duration_seconds_bucket series, and histogram_quantile(0.99, rate(llm_request_duration_seconds_bucket[5m])) in Prometheus returns a non-zero value after a few completion requests have been served.

Do's and Don'ts

Now that you have wired up automatic HTTP metrics, custom GenAI counters, and discipline-specific signals, the following rules keep the instrumentation queryable and the Prometheus server healthy as traffic scales.

Do's

1. ✓Do define explicit histogram buckets — the default buckets stop near 1 s and lose all p99 signal for LLM-backed routes; declare buckets that span 10 ms to 30 s.
2. ✓Do label custom metrics with model and endpoint — without these labels you cannot answer "which model on which route is driving cost" and the metric becomes shelfware.
3. ✓Do exclude /health and /metrics from instrumentation — internal high-frequency traffic skews request-rate and latency aggregates if left in.

Don'ts

1. ✗Don't use Summary metrics for latency in Kubernetes — they pre-compute quantiles per pod and cannot be aggregated across replicas, so Prometheus has no way to merge them.
2. ✗Don't put unbounded values in label names — labels like user_id or full request paths create one time series per value and will blow up Prometheus memory.
3. ✗Don't increment counters inside tight loops — use .inc(n) with the total once, not .inc() n times, especially for token counts in long completions.