Configure Prometheus monitoring for data services

Prometheus monitoring in Kubernetes is not merely about installing a Helm chart and hoping for the best. For production data infrastructure—PostgreSQL, Redis, and Kafka—you need a deliberate exporter topology that captures the metrics that actually predict failures before they cascade. This section walks through the architecture, deployment patterns, and programmatic validation of a complete monitoring stack using Prometheus exporters as sidecars and standalone deployments, ServiceMonitor custom resources for automatic scrape target discovery, and Python-based health verification scripts that confirm your entire observability pipeline is functioning end-to-end.

Introduction

When you run PostgreSQL, Redis, and Kafka on Kubernetes without dedicated exporter coverage, connection-pool exhaustion, consumer-group lag, and silent cache eviction become visible only after services fail. Exporters translate each service's native statistics into Prometheus-readable /metrics endpoints, and ServiceMonitor resources tell Prometheus where to scrape them. By the end of this lesson you will be able to deploy postgres-exporter, redis-exporter, and kafka-exporter with the correct topology, register them as Prometheus scrape targets via ServiceMonitor custom resources, and verify the observability pipeline is functioning before considering monitoring complete.

Key Terminology

• Exporter: A process that translates native service metrics into Prometheus exposition format, exposing them on an HTTP /metrics endpoint
• ServiceMonitor: A Prometheus Operator custom resource that declaratively defines scrape targets using Kubernetes label selectors
• Sidecar Pattern: Deploying an exporter container in the same Pod as the data service, sharing the Pod network namespace for localhost access
• pg_stat_activity: PostgreSQL system view that reports one row per server process, showing current query, state, and wait events
• pg_stat_user_tables: PostgreSQL system view exposing per-table statistics including sequential scans, index scans, and tuple counts
• pg_stat_bgwriter: PostgreSQL system view tracking background writer and checkpoint activity, critical for I/O tuning
• Consumer Group Lag: The difference between the latest offset in a Kafka partition and the committed offset of a consumer group, indicating processing backlog

Concepts

The diagram below shows the scrape topology: sidecar exporters share a Pod network namespace with their data service, while kafka-exporter runs as a standalone Deployment reaching brokers over the cluster network. Prometheus discovers all three through ServiceMonitor custom resources.

Critical Metrics to Alert On

Not all metrics warrant an alert. For data infrastructure, focus your Grafana dashboards and Prometheus alerting rules on the metrics that predict imminent failure:

1. Track connection saturation across all database services

   • pg_stat_activity_count approaching max_connections indicates PostgreSQL is nearing its connection ceiling—PgBouncer should be deployed as a connection pooler before this threshold is reached
   • redis_connected_clients exceeding expected baselines suggests a connection leak in application code
   • Alert at 80% of maximum capacity to allow time for scaling or connection pooling intervention

2. Monitor replication lag for data consistency guarantees

   • pg_stat_replication_pg_wal_lsn_diff measures bytes of WAL lag between primary and replicas—critical for read-after-write consistency in applications using read replicas
   • kafka_consumergroup_lag exceeding a per-topic threshold means consumers are falling behind producers, risking data staleness in downstream services
   • Set alerting thresholds relative to your SLA: if you promise 5-second data freshness, alert when lag exceeds 3 seconds

3. Watch I/O and memory pressure for capacity planning

   • pg_stat_bgwriter_buffers_checkpoint versus pg_stat_bgwriter_buffers_clean ratio indicates whether PostgreSQL is performing too many forced checkpoints—a sign that shared_buffers or checkpoint configuration needs tuning
   • redis_memory_used_bytes approaching maxmemory triggers eviction policies that may silently discard cache entries, degrading application performance without generating errors
   • Use HPA (Horizontal Pod Autoscaler) custom metrics to scale read replicas when query latency exceeds thresholds

4. Validate exporter availability as a meta-monitoring concern

   • The pg_up, redis_up, and kafka_brokers metrics serve as heartbeats for the exporters themselves—if these go to zero, your monitoring has a blind spot
   • Configure Prometheus absent() alerts to trigger when expected metric time series disappear entirely, catching exporter crashes that produce no error metrics

Code Walkthrough

Now that you understand which metrics signal imminent failure—connection saturation, replication lag, and I/O pressure—the next step is deploying the exporters that surface those metrics and wiring them into Prometheus.

The sidecar pattern is the preferred topology for database exporters in StatefulSet workloads. Running postgres-exporter as a sidecar within the same Pod as PostgreSQL eliminates network hops and couples the exporter lifecycle to the database instance it monitors. Redis-exporter follows the same sidecar approach. Kafka-exporter runs as a standalone Deployment because a single instance can reach an entire Kafka cluster through any broker's JMX endpoint. Prometheus discovers all three through ServiceMonitor custom resources watched by the Prometheus Operator.

The following script uses the kubernetes Python client to register ServiceMonitor resources for all three exporters in one pass:

Code snippetpython
1from kubernetes import client, config
2from kubernetes.client.rest import ApiException
3import logging
4
5logger = logging.getLogger(__name__)
6
7def create_service_monitor(name, namespace, port_name, interval, match_labels):
8    """Create a Prometheus ServiceMonitor custom resource."""
9    config.load_incluster_config()
10    api = client.CustomObjectsApi()
11
12    body = {
13        "apiVersion": "monitoring.coreos.com/v1",
14        "kind": "ServiceMonitor",
15        "metadata": {
16            "name": name,
17            "namespace": namespace,
18            "labels": {
19                "release": "prometheus",
20                "app.kubernetes.io/part-of": "data-monitoring",
21            },
22        },
23        "spec": {
24            "endpoints": [
25                {
26                    "port": port_name,
27                    "interval": interval,
28                    "path": "/metrics",
29                    "scrapeTimeout": "10s",
30                }
31            ],
32            "selector": {"matchLabels": match_labels},
33        },
34    }
35
36    try:
37        api.create_namespaced_custom_object(
38            group="monitoring.coreos.com",
39            version="v1",
40            namespace=namespace,
41            plural="servicemonitors",
42            body=body,
43        )
44        logger.info("Created ServiceMonitor %s in %s", name, namespace)
45        return True
46    except ApiException as exc:
47        if exc.status == 409:
48            logger.warning("ServiceMonitor %s already exists; skipping", name)
49            return False
50        raise
51
52EXPORTERS = [
53    {"name": "postgres-exporter", "namespace": "data",
54     "port_name": "pg-metrics", "interval": "30s",
55     "match_labels": {"app": "postgresql"}},
56    {"name": "redis-exporter", "namespace": "data",
57     "port_name": "redis-metrics", "interval": "15s",
58     "match_labels": {"app": "redis"}},
59    {"name": "kafka-exporter", "namespace": "monitoring",
60     "port_name": "kafka-metrics", "interval": "30s",
61     "match_labels": {"app": "kafka-exporter"}},
62]
63
64if __name__ == "__main__":
65    for exporter in EXPORTERS:
66        create_service_monitor(**exporter)

Once the ServiceMonitors are registered, confirm Prometheus is actually scraping each exporter by querying for a representative metric from each service—pg_stat_activity_count for PostgreSQL, redis_connected_clients for Redis, and kafka_consumergroup_lag for Kafka:

Code snippetpython
1import requests
2
3PROMETHEUS_URL = "http://prometheus.monitoring.svc.cluster.local:9090"
4
5EXPECTED_METRICS = [
6    "pg_stat_activity_count",
7    "redis_connected_clients",
8    "kafka_consumergroup_lag",
9]
10
11def metric_is_present(metric_name):
12    resp = requests.get(
13        f"{PROMETHEUS_URL}/api/v1/query",
14        params={"query": f"count({metric_name})"},
15        timeout=5,
16    )
17    resp.raise_for_status()
18    result = resp.json()["data"]["result"]
19    return len(result) > 0 and float(result[0]["value"][1]) > 0
20
21if __name__ == "__main__":
22    for metric in EXPECTED_METRICS:
23        ok = metric_is_present(metric)
24        status = "✓ present" if ok else "✗ MISSING"
25        print(f"{metric}: {status}")

Verify by running the validation script against your cluster and confirming each of the three metrics prints ✓ present; a ✗ MISSING result means the corresponding exporter is not yet scraped and the ServiceMonitor label selector or exporter Pod labels need adjustment.

Do's and Don'ts

Do's

1. ✓

   Do deploy postgres-exporter as a sidecar - Sidecar deployment eliminates network latency for metric collection and ensures the exporter's lifecycle matches the database Pod. If PostgreSQL restarts, the exporter restarts with it, preventing stale metric series.

2. ✓

   Do set distinct scrape intervals per exporter type - Database exporters executing SQL queries against system views should use 30-second intervals to avoid adding load. Lightweight exporters like redis-exporter can safely use 15-second intervals for faster anomaly detection.

3. ✓

   Do include _up metrics in every validation check - The pg_up, redis_up, and kafka_brokers gauge metrics are the fastest way to determine whether an exporter has lost connectivity to its target service. Always check these before investigating more complex metric absences.

Don'ts

1. ✗

   Don't expose exporter ports outside the cluster network - Exporter /metrics endpoints can leak sensitive operational data including query text from pg_stat_activity. Keep these on ClusterIP Services accessible only to Prometheus.

2. ✗

   Don't use a single kafka-exporter instance without resource limits - Kafka-exporter fetches consumer group offsets for every topic and partition. In clusters with thousands of partitions, this can consume significant memory. Set explicit resources.limits in the Deployment spec.

3. ✗

   Don't ignore scrape errors in Prometheus targets page - A target showing last scrape: error in the Prometheus UI means metrics are not being collected. These errors are silent—no alert fires because the metric series simply stops existing. Configure absent() alerts as a safety net.