Monitor gateway latency and token usage with Prometheus

Introduction

When you operate an LLM gateway serving multiple tenants, you have no reliable way to enforce SLOs or attribute token costs without real-time visibility into latency distributions, consumption totals, and cache effectiveness across every model and tenant pair. A single slow model or a runaway tenant can silently degrade the entire platform before anyone notices. By the end of this lesson, you'll be able to define the Prometheus metric instruments that capture gateway request latency and token usage, wire them into a FastAPI middleware that runs on every request, and confirm that Grafana can query and visualize the resulting per-tenant, per-model time series.

Key Terminology

• Prometheus Histogram — A metric instrument that accumulates observations into fixed bucket boundaries rather than storing raw samples, enabling percentile queries like P99; REQUEST_LATENCY uses buckets spanning 0.1 s to 30.0 s to cover the full range from semantic cache hits to cold large-model calls.
• Labeled Counter — A monotonically increasing metric instrument tagged with key-value dimensions; TOKEN_USAGE uses model, tenant_id, and token_type labels so Grafana can split prompt versus completion token costs at the per-tenant, per-model level without requiring separate metric names.
• Gauge — A metric instrument that can increase or decrease, used here as ACTIVE_REQUESTS to track in-flight requests per model; requires a try/finally guard in MetricsMiddleware.dispatch so the count decrements correctly even when downstream handlers raise exceptions.
• Bucket alignment — The practice of placing histogram bucket boundaries at the same numeric thresholds used in Alertmanager rules; the REQUEST_LATENCY bucket list includes 10.0 s precisely because the P99 alert fires at that threshold, and Prometheus percentile interpolation is most accurate when a boundary sits at the alert cutoff.
• token_type label — The "input" / "output" dimension on gateway_tokens_total that separates prompt tokens from completion tokens, enabling cost attribution because most model providers price input and output tokens at different rates.
• Cache hit ratio — A derived Grafana panel metric computed as rate(gateway_cache_hits_total[5m]) / (rate(gateway_cache_hits_total[5m]) + rate(gateway_cache_misses_total[5m])), measuring the fraction of requests served by the semantic cache rather than reaching the upstream model.

Concepts

Why Middleware Is the Right Observation Point

Attaching metric collection to individual route handlers creates blind spots: authentication failures, routing errors, and malformed requests never reach a handler, yet they still consume gateway capacity and affect the latency distribution that tenants experience. Placing MetricsMiddleware in the FastAPI middleware stack means every inbound request—successful, errored, or exception-raising—passes through a single observation point before any handler logic runs.

The try/finally block in dispatch is what makes this guarantee hold. Without it, an exception thrown by a downstream handler would skip the ACTIVE_REQUESTS.dec() call and the histogram observation, leaving the active-requests gauge drifting upward and latency records incomplete. The middleware pattern is not just a convenience; it is the only placement that produces accurate concurrency and latency data across all request outcomes (see Code Walkthrough).

Histogram Buckets Are a Correctness Decision, Not an Aesthetic One

Prometheus histograms do not store raw latency samples. They increment a count in the lowest bucket whose boundary exceeds the observed value, and histogram_quantile() interpolates the requested percentile by finding which bucket straddles the target rank. Accuracy is highest when a bucket boundary sits at or very near the threshold you query against.

This is why the REQUEST_LATENCY bucket list [0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0] includes 10.0 s exactly—the Alertmanager P99 rule fires at that threshold. If that bucket were absent, the interpolated P99 estimate at 10 s would span a much wider interval and could misfire the alert in either direction. The upper bound of 30.0 s captures cold calls to large models without producing an +Inf bucket overflow that would distort percentile calculations. Bucket selection should therefore be derived by working backward from your SLO thresholds, not chosen arbitrarily.

Labels Unlock Multi-Tenant Visibility and Cost Attribution

Every instrument declared in this lesson carries model and tenant_id labels. This combination allows a single gateway_tokens_total counter to serve both platform-wide aggregations (sum across all label values) and per-tenant cost breakdowns (filter by tenant_id) from identical underlying data. Adding the token_type label further separates prompt tokens from completion tokens, which is essential for accurate cost attribution because providers price them differently.

The CACHE_HITS and CACHE_MISSES counters exist as separate instruments rather than a single labeled counter so that the cache-hit-ratio panel can be expressed as a simple rate division in PromQL. Designing the full label schema before writing any middleware logic—as the lesson does—is intentional: the labels must anticipate the Grafana queries and Alertmanager selectors you will write later, and retrofitting labels onto a running counter resets its accumulated value (see Code Walkthrough).

Code Walkthrough

Now that you understand how the Request Metrics Middleware, Token Usage Collector, cache counters, and Prometheus endpoint each fit into the gateway request lifecycle, let's implement them.

The first step is declaring the metric instruments. LLM response times span several orders of magnitude—semantic cache hits return in milliseconds while cold calls to large models can take 20+ seconds—so the latency histogram needs buckets tuned for that range. Token counters carry a token_type label to separate input from output, which is what enables cost attribution by tenant in Grafana.

Code snippetpython
1from prometheus_client import Histogram, Counter, Gauge
2
3REQUEST_LATENCY = Histogram(
4    "gateway_request_duration_seconds",
5    "Request latency in seconds",
6    labelnames=["model", "tenant_id", "status"],
7    buckets=[0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0],
8)
9
10TOKEN_USAGE = Counter(
11    "gateway_tokens_total",
12    "Total tokens consumed",
13    labelnames=["model", "tenant_id", "token_type"],
14)
15
16CACHE_HITS = Counter(
17    "gateway_cache_hits_total",
18    "Semantic cache hits",
19    labelnames=["model"],
20)
21
22CACHE_MISSES = Counter(
23    "gateway_cache_misses_total",
24    "Semantic cache misses",
25    labelnames=["model"],
26)
27
28ACTIVE_REQUESTS = Gauge(
29    "gateway_active_requests",
30    "Currently in-flight requests",
31    labelnames=["model"],
32)

With the instruments declared, the middleware records observations on every request. The try/finally block guarantees the active-requests gauge decrements even when the downstream handler raises an exception—without it, the gauge drifts upward and produces misleading concurrency data that makes the gateway appear busier than it is.

Code snippetpython
1import time
2from fastapi import Request
3
4class MetricsMiddleware:
5    async def dispatch(self, request: Request, call_next):
6        model = "unknown"
7        tenant_id = getattr(request.state, "tenant_id", "unknown")
8
9        if request.method == "POST":
10            body = await request.json()
11            model = body.get("model", "unknown")
12
13        ACTIVE_REQUESTS.labels(model=model).inc()
14        start = time.monotonic()
15
16        try:
17            response = await call_next(request)
18            duration = time.monotonic() - start
19            status = str(response.status_code)
20
21            REQUEST_LATENCY.labels(
22                model=model, tenant_id=tenant_id, status=status
23            ).observe(duration)
24
25            if hasattr(response, "_token_usage"):
26                usage = response._token_usage
27                TOKEN_USAGE.labels(
28                    model=model, tenant_id=tenant_id, token_type="input"
29                ).inc(usage.get("prompt_tokens", 0))
30                TOKEN_USAGE.labels(
31                    model=model, tenant_id=tenant_id, token_type="output"
32                ).inc(usage.get("completion_tokens", 0))
33
34            return response
35        finally:
36            ACTIVE_REQUESTS.labels(model=model).dec()

The histogram buckets from 0.1 s to 30.0 s align directly with the Alertmanager rule that fires when P99 latency exceeds 10 s—every Alertmanager threshold needs a corresponding bucket boundary to be computed accurately from histogram data. The token_type label on TOKEN_USAGE lets Grafana split prompt versus completion token costs at the per-tenant, per-model level. The CACHE_HITS and CACHE_MISSES counters feed the cache-hit-ratio panel, which is computed as rate(gateway_cache_hits_total[5m]) / (rate(gateway_cache_hits_total[5m]) + rate(gateway_cache_misses_total[5m])).

Verify by sending a POST request to the gateway's /v1/chat/completions endpoint and then curling /metrics—you should see gateway_request_duration_seconds_bucket, gateway_tokens_total, and gateway_cache_hits_total each labeled with the model and tenant ID from your test request.

Do's and Don'ts

Now that you've wired the middleware, declared the histogram and counters, and verified the /metrics scrape, here are the patterns to follow and the pitfalls to avoid when extending this instrumentation.

Do's

1. ✓Do declare REQUEST_LATENCY histogram buckets that include every Alertmanager threshold you plan to enforce — gateway_request_duration_seconds uses [0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0] so the P99 > 10 s alert resolves to an accurate percentile; Prometheus interpolates across bucket spans, so a threshold that falls between two bucket edges produces a systematically imprecise estimate that can mask real SLO breaches.
2. ✓Do decrement ACTIVE_REQUESTS inside the finally block of MetricsMiddleware.dispatch — placing .labels(model=model).dec() there guarantees it runs even when call_next(request) raises an exception; any path that skips the decrement causes gateway_active_requests to drift upward, misrepresenting concurrency and making the gateway appear busier than it is.
3. ✓Do record separate TOKEN_USAGE increments for token_type="input" (from prompt_tokens) and token_type="output" (from completion_tokens) — the token_type label is what lets Grafana split prompt versus completion costs at the per-tenant, per-model level; without it, gateway_tokens_total collapses into a single unsplit total that cannot support cost attribution or budget enforcement across tenants.

Don'ts

1. ✗Don't rely on Prometheus's default histogram bucket set for gateway_request_duration_seconds — default buckets top out at 10 s, which cuts off the tail of cold LLM calls that routinely reach 20+ seconds; requests in the slow tail fall into the +Inf bucket and are invisible to percentile calculations, making P95/P99 SLOs for large models unenforceable.
2. ✗Don't omit tenant_id from the label sets on REQUEST_LATENCY and TOKEN_USAGE — without per-tenant cardinality in these instruments, a single runaway tenant's latency spike or token overconsumption is invisible in Grafana and indistinguishable from platform-wide load, defeating the gateway's primary cost-attribution and SLO-isolation guarantees.
3. ✗Don't skip curling /metrics after sending a test POST to /v1/chat/completions — the middleware reads model from the request body and tenant_id from request.state; if either extraction is broken, gateway_request_duration_seconds_bucket and gateway_tokens_total emit with "unknown" labels that Grafana queries return as empty time series with no error surfaced at instrumentation time.