Deploy SLA monitoring with Grafana dashboards

Introduction

In production, platform health is invisible until something breaks — and by then, SLA violations have already accumulated. Teams that deploy without a unified monitoring stack discover incidents through user complaints rather than automated alerts, and have no error-budget data to guide reliability work. This lesson covers deploying the full observability stack — Prometheus, Alertmanager, Grafana, plus the custom SLO tracker and status page services — as a cohesive, Helm-managed unit. By the end, you'll be able to generate environment-specific Helm values, wire alert routing to PagerDuty and Slack, and provision Grafana dashboards as code.

Key Terminology

• generate_monitoring_values — A Python function that produces an environment-specific configuration dict for the entire observability stack in one pass, setting Prometheus retention and storage, Alertmanager receiver routing, Grafana datasource and dashboard provisioning, and SLO tracker replica count based on the environment argument.
• Alertmanager routing tree — A nested configuration structure inside the alertmanager.config.route block that directs alerts to different receivers based on the severity label: critical alerts page the on-call engineer via PagerDuty, warning alerts post to #platform-oncall, and unmatched alerts fall through to the default #platform-alerts Slack channel.
• Dashboard provisioning (dashboards as code) — The practice of tracking Grafana dashboard definitions in a Kubernetes ConfigMap (grafana-dashboards) referenced by the dashboardProviders block, so ArgoCD syncs dashboards to the cluster on every deploy rather than requiring manual UI imports.
• Scrape interval and evaluation interval — The 15-second cadences at which Prometheus collects metrics from targets (scrapeInterval) and re-evaluates alert rules (evaluationInterval); both are fixed across environments in generate_monitoring_values to guarantee consistent alert timing regardless of environment size.
• Stdin values injection — The pattern used in deploy_monitoring_stack of streaming the Helm values YAML through /dev/stdin via subprocess.run rather than writing to a temporary file, preventing secrets such as PAGERDUTY_KEY from ever touching disk.
• Environment-specific sizing — The conditional logic inside generate_monitoring_values that sets Prometheus retention to 30 days and storage to 100 GiB on production versus 7 days and 20 GiB on staging, keeping cloud costs proportional to the reliability requirements of each environment.

Concepts

Treating the Observability Stack as a Single Deployable Unit

The most important mental model shift in this lesson is deploying Prometheus, Alertmanager, Grafana, the SLO tracker, and the status page as one cohesive Helm release rather than as independently managed services. When these components are installed separately, their configurations can silently diverge: a Prometheus scrapeInterval that doesn't match the Alertmanager evaluationInterval weakens alert fidelity without producing any visible error, and a Grafana datasource pointing at a stale Prometheus URL renders empty panels with no indication of misconfiguration. Generating all values in a single function call — generate_monitoring_values — and applying them atomically via helm upgrade --install ensures that every component starts from a mutually consistent state. If the deploy fails partway through, Helm rolls back the entire release; there's no half-configured cluster to debug (see Code Walkthrough).

Alert Severity Routing as a Noise Isolation Contract

The three-tier Alertmanager routing tree is not organizational tidiness — it is a contract that keeps high-signal alerts visible. The failure mode it prevents is alert fatigue: when warning-level threshold crossings and genuine critical outages land in the same notification channel, engineers begin treating all alerts as noise and miss the ones that matter. The routing tree implements three distinct noise floors. Critical alerts — a service down, an SLA breach in progress — interrupt the on-call engineer immediately via PagerDuty. Warning alerts post to #platform-oncall for team awareness without an interrupt. Everything else flows to #platform-alerts as a low-priority information stream that can be reviewed asynchronously. This three-way separation means each severity tier has exactly one destination, and on-call engineers always know which surface to watch for genuine emergencies versus ambient health signals.

Dashboards as Code via ConfigMaps and ArgoCD

The traditional Grafana workflow requires engineers to import dashboards manually through the UI — a process that is not version-controlled, not reproducible across clusters, and evaporates whenever a namespace is torn down. The dashboardProviders configuration block solves this by pointing Grafana at a ConfigMap named grafana-dashboards. ArgoCD syncs that ConfigMap from the repository on every deploy, which means the dashboards visible in the Grafana UI are always exactly what the codebase defines — no manual import step, no config drift between production and staging, no "that dashboard only existed on the old cluster." Because the ConfigMap is a standard Kubernetes resource, it participates in the same GitOps review and rollback workflow as application code: a dashboard regression is a PR revert, not a manual UI edit (see Code Walkthrough).

Environment-Specific Sizing Without Separate Files

A single generate_monitoring_values function emits meaningfully different configurations for production and staging by conditionalizing on the environment argument. Production receives 30-day Prometheus retention and 100 GiB of storage — sized to support a post-incident root-cause analysis days after an event — while staging gets 7-day retention and 20 GiB, keeping costs proportional to value delivered. The SLO tracker likewise runs two replicas in production for availability and one in staging. This approach avoids the common alternative of maintaining separate values-prod.yaml and values-staging.yaml files, which inevitably drift out of sync as teams update one and forget the other. All environment differences are explicit Python conditionals in one function, so a reviewer can see exactly what changes between environments in a single diff.

Code Walkthrough

Now that you understand the architecture — Prometheus scraping metrics every 15 seconds, Alertmanager routing by severity, and Grafana visualizing SLO compliance — the next step is generating the Helm values that wire these components together into a single deployable unit.

The generate_monitoring_values function produces environment-specific configuration for the entire stack in one pass. Prometheus receives retention and storage settings sized for production versus staging — 30-day retention and 100 GiB on production, 7-day and 20 GiB on staging — along with fixed scrape and evaluation intervals. Alertmanager gets a three-tier routing tree. Grafana is configured with pre-provisioned dashboard ConfigMaps and a default Prometheus datasource, so dashboards are code-tracked and deployed alongside the application on every ArgoCD sync.

Code snippetpython
1def generate_monitoring_values(environment: str) -> dict:
2    is_prod = environment == "production"
3    return {
4        "prometheus": {
5            "retention": "30d" if is_prod else "7d",
6            "storage": {"size": "100Gi" if is_prod else "20Gi"},
7            "scrapeInterval": "15s",
8            "evaluationInterval": "15s",
9            "resources": {
10                "requests": {"cpu": "500m", "memory": "2Gi"},
11                "limits": {"cpu": "2000m", "memory": "8Gi"},
12            },
13        },
14        "alertmanager": {
15            "config": {
16                "route": {
17                    "receiver": "slack-default",
18                    "routes": [
19                        {"match": {"severity": "critical"}, "receiver": "pagerduty"},
20                        {"match": {"severity": "warning"}, "receiver": "slack-oncall"},
21                    ],
22                },
23                "receivers": [
24                    {
25                        "name": "slack-default",
26                        "slack_configs": [{"channel": "#platform-alerts"}],
27                    },
28                    {
29                        "name": "pagerduty",
30                        "pagerduty_configs": [{"service_key": "$(PAGERDUTY_KEY)"}],
31                    },
32                    {
33                        "name": "slack-oncall",
34                        "slack_configs": [{"channel": "#platform-oncall"}],
35                    },
36                ],
37            },
38        },
39        "grafana": {
40            "dashboardProviders": {
41                "enabled": True,
42                "configMapName": "grafana-dashboards",
43            },
44            "datasources": [
45                {
46                    "name": "Prometheus",
47                    "type": "prometheus",
48                    "url": "http://prometheus:9090",
49                    "isDefault": True,
50                },
51            ],
52        },
53        "sloTracker": {
54            "replicaCount": 2 if is_prod else 1,
55        },
56    }

The Alertmanager routing tree implements the severity ladder from the architecture overview: critical alerts route to PagerDuty to page the on-call engineer immediately, warning alerts post to #platform-oncall in Slack, and the default receiver forwards everything else to #platform-alerts. This three-tier separation ensures that noisy informational alerts do not compete with genuine outages for attention. Grafana's dashboardProviders block points at a ConfigMap named grafana-dashboards; ArgoCD syncs that ConfigMap from the repository on each deploy, so the dashboards in the UI always match the codebase.

Once the values dict is ready, pass it directly to Helm's stdin to avoid writing secrets to disk:

Code snippetpython
1import subprocess
2import yaml
3
4def deploy_monitoring_stack(environment: str) -> None:
5    values = generate_monitoring_values(environment)
6    subprocess.run(
7        [
8            "helm", "upgrade", "--install", "monitoring",
9            "prometheus-community/kube-prometheus-stack",
10            "--namespace", "monitoring",
11            "--create-namespace",
12            "--values", "/dev/stdin",
13        ],
14        input=yaml.dump(values).encode(),
15        check=True,
16    )

You'll know it works when kubectl get pods -n monitoring shows Prometheus, Alertmanager, Grafana, the SLO tracker, and the status page all in Running state, and the Grafana UI at the configured ingress displays the pre-provisioned dashboards without any manual import steps.

Do's and Don'ts

Having walked through the values generator, the routing tree, and the dashboard ConfigMap, the remaining decisions are operational. The rules below distil the deploy-time choices that determine whether the monitoring stack stays trustworthy in production.

Do's

1. ✓Do size Prometheus retention and storage per environment using generate_monitoring_values — production's 30-day/100 GiB settings protect long-term SLO trend analysis, while staging's 7-day/20 GiB keeps costs low; hard-coding a single profile across environments either wastes resources or silently discards the historical data error-budget calculations depend on.
2. ✓Do pipe Helm values through /dev/stdin rather than writing them to a file — the deploy_monitoring_stack function passes the YAML-encoded values dict via subprocess.run(input=...) specifically to prevent secrets like PAGERDUTY_KEY from being written to disk, where they could be captured by logging or CI artifact archiving.
3. ✓Do provision Grafana dashboards as code via the dashboardProviders ConfigMap — setting configMapName: grafana-dashboards with enabled: True means ArgoCD syncs the dashboards on every deploy so the Grafana UI always matches the repository; manually imported dashboards drift silently and disappear on pod restarts.

Don'ts

1. ✗Don't collapse the Alertmanager routing tree to a single receiver — the three-tier separation (critical → PagerDuty, warning → #platform-oncall, default → #platform-alerts) is what prevents low-signal informational noise from burying a genuine outage; routing all severities to one channel means on-call engineers stop reading it.
2. ✗Don't deploy the monitoring stack components independently instead of as a unified Helm release — helm upgrade --install monitoring prometheus-community/kube-prometheus-stack wires Prometheus, Alertmanager, Grafana, the SLO tracker, and the status page together with consistent label selectors and service discovery; deploying them separately breaks scrape target discovery and leaves the stack partially observable until all components share the same release.
3. ✗Don't treat a successful helm upgrade as confirmation the stack is healthy — the real verification is kubectl get pods -n monitoring showing every component in Running state and the Grafana ingress serving pre-provisioned dashboards without a manual import; a Helm exit code of 0 only means the API accepted the manifests, not that Prometheus is scraping or that Alertmanager has resolved its receiver configs.