Monitor Kafka with consumer lag metrics

Introduction

When you ship AI inference or embedding pipelines on Kafka, lag is the metric that decides whether users see fresh answers or stale ones — and it is the one most often misinterpreted. Lag is the per-partition difference between producer-written offsets and consumer-committed offsets — how far behind real-time your pipeline is. Get it wrong and a 4,000-record backlog on inference-requests becomes a 20-second user-visible latency spike that no APM dashboard surfaces, while RAG quietly returns stale context because embeddings-ingest fell minutes behind. By the end you will be able to measure lag accurately, distinguish absolute lag from lag-time, alert on what users feel, and run the slow-consumer response runbook (scale up, rewind, replay).

Key Terminology

• Log-end-offset (LEO) — the next offset producers will write to a partition; the moving right edge of the log that every lag calculation is measured against.
• Committed offset — the offset a consumer group has acknowledged and persisted to __consumer_offsets; the left edge of unread work and the value that drives lag.
• Absolute lag — LEO - committed_offset per partition, summed across assignments for the group; necessary but not sufficient because record count alone hides whether 1,000 records means 100 seconds or 10 ms of delay.
• Lag-time — the wall-clock age of the next unconsumed record; the metric users actually feel and the right SLO target for user-facing topics like inference-requests.
• Rebalance storm — repeated, rapid flipping of consumer-group state that halts consumption during each rebalance; a top cause of growing lag even when capacity looks healthy.

Concepts

What lag actually measures

For each partition the broker tracks two cursors. The log-end-offset (LEO) is the next offset producers will write. The committed offset is what the group has acknowledged, persisted to __consumer_offsets. Absolute lag is LEO - committed_offset; the group's total lag sums across assigned partitions.

That integer is necessary but not sufficient: 1,000 records on a 10 rec/s topic is 100 seconds behind; the same on a 100,000 rec/s topic is 10 ms behind — invisible. The metric users feel is lag-time: the wall-clock age of the next unconsumed record.

Why lag is topic-specific for AI pipelines

Lag tolerance is set by what the topic feeds. embeddings-ingest feeds the vector index — lag means new docs are not yet searchable, so RAG returns stale context. inference-requests queues user requests for GPUs — a 4,000-record lag at 200 req/s is a 20-second user-visible latency spike. SLOs follow: seconds for inference-requests, minutes for embeddings-ingest, hours for control-plane topics like model-events.

Uneven lag across partitions of the same topic is itself diagnostic — it points at hot keys or a failed instance, not raw capacity.

The four causes of chronic lag

When lag is consistently high or growing, the cause is almost always one of four things:

1. Too few consumer instances. 12 partitions, 4 consumers, 100 ms per message = peak 120 rec/s. Producer rate above that grows lag forever. Fix: scale to the partition count, then add partitions carefully (new partitions break key-based ordering).
2. Slow per-message processing. The most common AI cause: an embedding call that took 50 ms in dev now takes 800 ms because the model server is overloaded. Profile, externalize blocking calls, or batch.
3. Downstream backpressure. Consumer is fast but writes to a rate-limited vector DB; the poll loop blocks. Fix at the downstream layer — adding consumers multiplies pressure on the bottleneck.
4. Rebalance storms. Consumers join and leave faster than the coordinator can finish a rebalance; no consumption happens during one. Causes: session.timeout.ms too low, processing exceeding max.poll.interval.ms, or pod churn (HPA flap, OOMKills, liveness failures).

Two alert classes that map to user pain

Absolute lag catches unbounded growth — producer rate exceeds consumer rate. Lag-time catches the case where record count is small but the topic is so slow that small lag still means minutes of delay. A third signal, member-count flap, catches rebalance storms before lag visibly grows. The Prometheus rules in Code Walkthrough encode all three.

Slow-consumer response runbook

When KafkaConsumerLagRecords or KafkaConsumerLagTime pages, work this checklist top-down:

1. Confirm and classify (60 s). Run kafka-consumer-groups.sh --describe --group $GROUP --state. Concentrated per-partition lag points at instance failure or hot keys; uniform lag points at a producer surge or downstream bottleneck.
2. Scale up consumers (never above partition count — extras idle): kubectl scale deploy/embedding-worker --replicas=<partitions>. If lag keeps growing past replicas == partitions, you have a per-message-cost problem.
3. Rewind offset (only when lag is unrecoverable AND staleness is acceptable): stop consumers, --reset-offsets --to-latest --execute, restart. This drops records on the floor — document what was skipped and what downstream artifacts need backfill.
4. Replay from a known-good offset for wrong-offset bugs: create a parallel group reading from a chosen timestamp, run the fixed consumer there, verify, cut over. Never replay into the live group.

Lag is also often downstream of broker problems — keep under-replicated partitions, broker disk usage, and a healthy __consumer_offsets on the same dashboard.

Code Walkthrough

Building on the lag-time and rebalance concepts above, this section walks through two concrete implementations: the lag collector (computing both absolute lag and lag-time without poisoning the production group) and the alert rules (absolute lag, lag-time, rebalance storm) that fire from the resulting metrics.

Code snippetpython
1from confluent_kafka import Consumer, TopicPartition
2from confluent_kafka.admin import AdminClient, ConsumerGroupTopicPartitions
3import time
4
5def collect_group_lag(bootstrap, group_id):
6    admin = AdminClient({"bootstrap.servers": bootstrap})
7    c = Consumer({"bootstrap.servers": bootstrap,
8                  "group.id": f"lag-probe-{int(time.time())}",
9                  "enable.auto.commit": False})
10    resp = admin.list_consumer_group_offsets(
11        [ConsumerGroupTopicPartitions(group_id)])[group_id].result(timeout=15)
12    out = []
13    for tp in resp.topic_partitions:
14        _, log_end = c.get_watermark_offsets(
15            TopicPartition(tp.topic, tp.partition), timeout=5.0, cached=False)
16        lag = max(0, log_end - tp.offset)
17        lag_s = None
18        if lag > 0:
19            c.assign([TopicPartition(tp.topic, tp.partition, tp.offset)])
20            msg = c.poll(timeout=2.0)
21            if msg and not msg.error() and msg.timestamp()[0] != 0:
22                lag_s = max(0.0, time.time() - msg.timestamp()[1] / 1000.0)
23        out.append((tp.topic, tp.partition, tp.offset, log_end, lag, lag_s))
24    c.close()
25    return out

The probe uses a unique group.id so it never joins the production group's rebalance, and enable.auto.commit=False so it cannot write to __consumer_offsets. cached=False forces a metadata round-trip — cached watermarks turn real alerts into false negatives. In production, prefer kafka_exporter as a sidecar exposing the same family of metrics; reach for this collector when you need lag-time alongside absolute lag in one scrape.

Code snippetyaml
1groups:
2  - name: kafka-consumer-lag
3    rules:
4      - alert: KafkaConsumerLagRecords
5        expr: sum by (consumergroup, topic) (kafka_consumergroup_lag) > 5000
6        for: 2m
7        labels: { severity: page, topic_class: inference }
8      - alert: KafkaConsumerLagTime
9        expr: max by (consumergroup, topic) (kafka_consumergroup_lag_seconds) > 30
10        for: 1m
11        labels: { severity: page, topic_class: inference }
12      - alert: KafkaRebalanceStorm
13        expr: rate(kafka_consumergroup_members[5m]) > 0.05
14        for: 3m
15        labels: { severity: page }

The three rules map one-to-one onto the alert classes from Concepts: record-count growth, lag-time SLO breach, and member-count flap. Always use for: — a 30-second spike was self-healing; a two-minute one is an incident.

You'll know it works when (a) a synthetic producer-pause makes KafkaConsumerLagRecords fire within ~2 minutes and clears after consumers catch up, (b) KafkaConsumerLagTime fires independently of record count on a slow topic, and (c) bouncing a consumer pod three times in five minutes trips KafkaRebalanceStorm.

Do's and Don'ts

Do's

1. ✓Do alert on lag-time, not just record count — users feel time, not records; page humans only on lag-time SLO breach for user-facing topics.
2. ✓Do use a separate group.id for lag probing — sharing the production coordinator session pollutes rebalances and can re-commit offsets.
3. ✓Do treat kafka_consumergroup_members flap as an incident — three drops in five minutes is a rebalance storm and consumption is halted during each.

Don'ts

1. ✗Don't scale consumers above partition count — extras sit idle; if lag keeps growing at replicas == partitions, the bottleneck is per-message cost, not parallelism.
2. ✗Don't replay into the live consumer group — it has already moved past those offsets; spin up a parallel group, verify, then cut over.
3. ✗Don't ignore __consumer_offsets health — if it goes under-replicated, every group's commit can stall and your lag metrics start lying.