Instrument all SLIs with Prometheus metrics and Langfuse traces

Introduction

Engineers often instrument latency and quality first, then discover during a load spike that they cannot answer the most basic capacity question: how many requests is the system actually serving per second, how many are we rejecting, and how close are we to saturation? A latency dashboard that looks healthy can hide a system that is silently shedding load — rejected requests never enter the latency histogram, so p99 stays flat while real users get 429s. In LLM-powered systems the throughput picture is doubly subtle, because raw request rate hides the fact that one long-context generation occupies a worker far longer than a short one. By the end of this lesson you'll be able to expose every throughput SLI — request rate (total/successful/rejected), tokens-per-second generation throughput, concurrent in-flight requests, and queue depth — as a correctly-typed Prometheus metric, wired through a single ThroughputSLICollector that your request middleware calls at well-defined lifecycle points.

Key Terminology

• Requests Per Second (RPS) — the rate at which requests arrive and complete, decomposed into total, successful, and rejected streams; instrumented as monotonic Counters and turned into a rate with PromQL rate(). The split matters because a flat success rate with a rising rejection rate means you are at capacity.
• Tokens-Per-Second (TPS) — the generation throughput of a single completion (output tokens ÷ generation seconds); the LLM-specific throughput SLI that raw RPS cannot capture, since one request may emit 20 tokens and another 2000. Instrumented as a Histogram so you can read p50/p99 generation speed.
• In-Flight Requests — the instantaneous count of requests currently being processed; a Gauge that is incremented when a request starts and decremented when it ends. It is the truest real-time saturation signal for a fixed worker pool.
• Queue Depth / Saturation — the number of requests waiting for a free worker; a Gauge whose sustained rise is the leading indicator that arrival rate has overtaken service rate, before latency or rejections visibly degrade.
• Counter vs. Gauge vs. Histogram — the three core Prometheus metric types. Counter only goes up (event totals → rates); Gauge moves up and down (point-in-time state); Histogram buckets observations (value distributions → percentiles). Choosing wrong silently destroys the signal.

Concepts

Why each throughput SLI needs a specific Prometheus metric type

Picking the wrong metric type does not raise an error — it quietly produces meaningless data, which is worse. Each throughput SLI has exactly one type that fits its mathematical shape.

• RPS → Counter. Request totals are monotonic events: each request happens once and the cumulative count never decreases. A Counter is the only type that survives a process restart cleanly (it resets to 0, and rate() detects the reset) and the only type rate(genai_requests_total[1m]) can turn into a true per-second arrival rate. Splitting into total, successful, and rejected Counters lets you compute the rejection ratio — rate(rejected[5m]) / rate(total[5m]) — which is the canonical "are we at capacity" signal.
• In-flight & queue depth → Gauge. These are instantaneous state, not events. The number of requests currently in flight goes up and down continuously, and at any scrape you want the value right now, not an accumulation. A Gauge is the only type that models a quantity that can decrease. Note the tradeoff: a Gauge drops the values between scrapes, which is exactly right for "current saturation" and exactly wrong for "how many requests happened" — that is why RPS must never be a Gauge.
• Tokens-per-second → Histogram. Generation throughput varies per request and you care about the distribution: a p50 of 80 tok/s with a p99 of 12 tok/s tells you tail requests (long context, large outputs) are starving. Only a Histogram preserves the bucket distribution that histogram_quantile() needs. A Gauge would keep only the last request's speed; a Counter cannot express "speed" at all.

Reading capacity and saturation from these signals

Instrumented together, these four SLIs let you reason about capacity directly. In-flight requests approaching your worker-pool size means you are running hot; a rising queue depth while in-flight is pinned at the ceiling means arrival rate has overtaken service rate and you are now accumulating latency that the latency histogram will only show seconds later. The rejection-rate Counter is the hard backstop: once the queue is full, new requests are rejected, and rate(rejected_total[1m]) becomes non-zero. The leading indicator (queue depth) fires before the lagging one (rejections), which is why both are instrumented rather than just the one users feel.

The middleware increments the in-flight Gauge and the total Counter the instant a request enters, calls the LLM, then on completion observes tokens-per-second on the Histogram, bumps the success or rejection Counter, and decrements the in-flight Gauge. Every throughput SLI is emitted from this one boundary, so the registry is always internally consistent.

One collector, three lifecycle hooks

All four SLIs are wired through a single ThroughputSLICollector that the middleware calls at three lifecycle points: request start (increment in-flight Gauge, increment total Counter, set queue depth), request completion (observe tokens-per-second Histogram), and request end (increment the successful or rejected Counter, decrement in-flight Gauge). Keeping this logic out of endpoint handlers means a new throughput SLI is added by editing the collector alone, and the inc/dec pairing for the in-flight Gauge lives in exactly one place — which is the only way to guarantee it never leaks a phantom in-flight count when a request errors (see Code Walkthrough).

Code Walkthrough

Now that you have the metric-type mapping and the lifecycle model, the two classes below turn that design into runnable code. ThroughputSLIRegistry registers every throughput metric with the correct type and GenAI-tuned buckets; ThroughputSLICollector drives them from the three middleware hooks.

Code snippetpython
1from prometheus_client import Counter, Gauge, Histogram, CollectorRegistry
2
3class ThroughputSLIRegistry:
4    """Central registry for all GenAI throughput SLI Prometheus metrics."""
5
6    def __init__(self, registry: CollectorRegistry = None):
7        self.registry = registry or CollectorRegistry()
8        labels = ["provider", "model", "endpoint"]
9
10        # RPS — Counters are monotonic; rate() turns them into per-second arrival
11        # and rejection rates. Split so rejection ratio = rejected / total.
12        self.requests_total = Counter(
13            "genai_requests_total", "Total requests received",
14            labelnames=labels, registry=self.registry,
15        )
16        self.requests_successful_total = Counter(
17            "genai_requests_successful_total", "Requests that completed successfully",
18            labelnames=labels, registry=self.registry,
19        )
20        self.requests_rejected_total = Counter(
21            "genai_requests_rejected_total", "Requests rejected (queue full / overload)",
22            labelnames=labels, registry=self.registry,
23        )
24
25        # Tokens-per-second — Histogram preserves the distribution so
26        # histogram_quantile() can report p50/p99 generation speed.
27        self.tokens_per_second = Histogram(
28            "genai_tokens_per_second", "Output-token generation throughput",
29            labelnames=labels,
30            buckets=(5, 10, 20, 30, 50, 75, 100, 125, 150),
31            registry=self.registry,
32        )
33
34        # In-flight & queue depth — Gauges: instantaneous state that goes up and
35        # down. Never Counters; these must be able to decrease.
36        self.in_flight_requests = Gauge(
37            "genai_in_flight_requests", "Requests currently being processed",
38            labelnames=labels, registry=self.registry,
39        )
40        self.queue_depth = Gauge(
41            "genai_queue_depth", "Requests waiting for a free worker",
42            labelnames=labels, registry=self.registry,
43        )

Every type is deliberate: the three requests_*_total Counters feed rate(...) for arrival and rejection rates; tokens_per_second is a Histogram with buckets spanning realistic LLM generation speeds (5–150 tok/s) so neither slow tail requests nor fast short ones collapse into one bin; in_flight_requests and queue_depth are Gauges because they model a quantity that decreases. The shared ["provider", "model", "endpoint"] label set lets Grafana break every throughput dimension down by model on one dashboard.

Code snippetpython
1import time
2
3class ThroughputSLICollector:
4    """Instruments throughput SLIs from the request middleware lifecycle."""
5
6    def __init__(self, prom: ThroughputSLIRegistry):
7        self.prom = prom            # ThroughputSLIRegistry instance
8        self._active: dict = {}
9
10    def on_request_start(self, endpoint: str, provider: str, model: str,
11                         queue_depth: int) -> str:
12        lbl = (provider, model, endpoint)
13        request_id = f"{endpoint}-{time.perf_counter_ns()}"
14        self._active[request_id] = {"labels": lbl, "t0": time.perf_counter()}
15
16        # Request entered: count it, mark one more in flight, record saturation.
17        self.prom.requests_total.labels(*lbl).inc()
18        self.prom.in_flight_requests.labels(*lbl).inc()
19        self.prom.queue_depth.labels(*lbl).set(queue_depth)
20        return request_id
21
22    def on_llm_complete(self, request_id: str, output_tokens: int) -> None:
23        ctx = self._active[request_id]
24        lbl, t0 = ctx["labels"], ctx["t0"]
25        gen_seconds = max(time.perf_counter() - t0, 1e-6)
26
27        # Generation throughput for THIS request → Histogram observation.
28        tps = output_tokens / gen_seconds
29        self.prom.tokens_per_second.labels(*lbl).observe(tps)
30
31    def on_request_end(self, request_id: str, rejected: bool = False) -> None:
32        ctx = self._active.pop(request_id)
33        lbl = ctx["labels"]
34
35        # Terminal outcome → success or rejection Counter.
36        if rejected:
37            self.prom.requests_rejected_total.labels(*lbl).inc()
38        else:
39            self.prom.requests_successful_total.labels(*lbl).inc()
40
41        # Always decrement in-flight — even on the error path — or the Gauge
42        # leaks a phantom count and saturation reads permanently high.
43        self.prom.in_flight_requests.labels(*lbl).dec()

on_request_start is the entry hook: it bumps the total Counter, increments the in-flight Gauge, and snapshots queue depth. on_llm_complete derives tokens-per-second for that single request and observes it on the Histogram. on_request_end is the terminal hook — it increments either the successful or the rejected Counter and always decrements the in-flight Gauge. Pairing the inc in on_request_start with the dec in on_request_end in one collector is what guarantees in-flight is never double-counted or leaked, so the saturation signal stays trustworthy under errors and rejections alike.

Verify by instantiating ThroughputSLIRegistry with a fresh CollectorRegistry, calling on_request_start, sleeping briefly, then on_llm_complete(request_id, output_tokens=100) and on_request_end(request_id). Confirm generate_latest(registry) shows genai_requests_total and genai_requests_successful_total at 1, genai_in_flight_requests back at 0, and a single genai_tokens_per_second observation roughly equal to 100 / elapsed_seconds.

Do's and Don'ts

Now that you have the collector and its lifecycle hooks in code, these rules keep your throughput instrumentation honest under real load.

Do's

1. ✓Do split request totals into total, successful, and rejected Counters — a single requests_total Counter tells you arrival rate but hides shedding. With all three, rate(rejected[5m]) / rate(total[5m]) gives the rejection ratio, the single clearest signal that you have hit capacity. Rejected requests never appear in your latency histogram, so without this split a saturated system can look perfectly healthy.
2. ✓Do pair every in-flight Gauge inc() with a dec() on all exit paths — including the error and rejection paths. Increment in on_request_start, decrement in on_request_end, and keep both in one collector. A missed dec() leaks a phantom in-flight count that never clears, so your saturation Gauge ratchets upward and eventually triggers false overload alerts.
3. ✓Do choose tokens-per-second Histogram buckets that span real generation speeds — buckets from ~5 to ~150 tok/s resolve the difference between a healthy p50 and a starving p99 tail. Default HTTP-tuned buckets collapse every generation into one bin and make histogram_quantile() on TPS meaningless.

Don'ts

1. ✗Don't use a Gauge for request totals or a Counter for in-flight requests — a Gauge drops the events between scrapes, so two requests arriving inside one scrape interval count as one, destroying your RPS. A Counter cannot decrease, so it can never represent in-flight count. Counter for monotonic totals, Gauge for instantaneous state — the mapping is not interchangeable.
2. ✗Don't infer LLM throughput from RPS alone — one request may emit 20 tokens and another 2000, so a flat request rate can hide a collapsing generation throughput. Always observe tokens-per-second on its own Histogram; it is the only signal that catches a model getting slower per token while request count looks stable.
3. ✗Don't put high-cardinality labels like request_id or user_id on throughput metrics — every unique combination is a new time series, so 10k requests/hour means 10k series per metric and overwhelmed Prometheus storage. Keep labels to bounded dimensions like provider, model, and endpoint; use a tracing backend for per-request drill-down.