Create Grafana dashboards for API monitoring

Introduction

When you're paged at 02:00, your Grafana dashboard must answer three questions in under five seconds: Is the system actually broken?, Where is the pain concentrated?, and What changed recently? Teams that ship dashboards as scratch pads end up with wallpaper — rows of pretty charts nobody trusts when traffic to a GenAI API spikes and every dropped request is a wasted GPU second. By the end of this lesson you'll be able to design a golden-signals dashboard for an API service, write the PromQL behind each panel, and store the dashboard as code so it survives the next reorganisation.

Key Terminology

• Golden signals — latency, traffic, errors, and saturation; the four measurements that together tell you whether a request-serving API is healthy, drawn from Google SRE practice and the first thing every dashboard surfaces.
• PromQL — the query language Grafana panels use against Prometheus. Every panel is a one-line PromQL expression; getting the rate windows and aggregation labels right is what separates a useful dashboard from a misleading one.
• histogram_quantile — the PromQL function that converts a Prometheus histogram's _bucket series into a percentile like P95 latency. Averaging _sum / _count instead is the single most common dashboard lie.
• Template variable — a dropdown at the top of a dashboard (e.g. $service, $route) that re-parameterises every panel. Variables let one dashboard serve every service and let an incident commander pivot from "Europe is degraded" to a single route in three clicks.
• Dashboard-as-code — storing the dashboard JSON in git and rendering it with Grafonnet or Terraform instead of editing in the UI. Eliminates drift, gives you review and history, and is the only way to keep alert thresholds and panel definitions in sync across environments.

Concepts

The three-tier layout

Discipline starts with a fixed layout that every API service inherits, so an on-call engineer jumping between services never has to relearn where information lives. The top row carries the four golden signals at full width — they answer Is anything wrong?. The middle band breaks the same signals down per HTTP route, because a 5xx spike is meaningless until you know which endpoint is bleeding. The bottom band exposes dependencies (database, cache, upstream LLM provider) — in GenAI APIs that is where most outages actually originate.

Golden signals as PromQL

Every tier-one panel is a one-liner against Prometheus. P95 latency uses histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))) — always group by le and the dimensions you want to keep, because dropping le collapses the histogram into nonsense. Error rate is sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100, with numerator and denominator on the same window so the ratio is dimensionally consistent. Traffic is sum by (route, method) (rate(http_requests_total[1m])), rendered as stacked bars — a sudden flat line means an upstream load balancer stopped sending traffic. Saturation is per-resource: CPU and memory per pod, DB pool usage, and on async Python services, event-loop lag — the canary that turns red while CPU still looks fine.

Variables, annotations, and links

A dashboard becomes a navigation surface when its header exposes cascading variables ($datasource, $namespace, $service, $route) so the incident commander can drill from region to route in three clicks. Overlaying deploy annotations turns "latency spiked at 14:32" into "the v2.41 canary went bad." And every panel exposes a panel-link that round-trips into traces and logs preserving the current variables and time range — the on-call clicks the latency panel and lands on a relevant trace within seconds. These three patterns compound; they are the difference between MTTR measured in minutes and in tens of minutes (see Code Walkthrough).

Dashboard-as-code

UI-edited dashboards drift, lose history, and disappear when someone clicks the wrong button. The cure is to keep the JSON in git, generate it from a small Python or Jsonnet program, and deploy with Terraform's Grafana provider or grafana-operator. Every panel flows through one helper so units, thresholds, and the runbook URL are enforced centrally — and CI can post a screenshot diff against main so reviewers see what changed visually, not just textually.

Code Walkthrough

Now that you understand the three-tier layout and the PromQL behind each golden signal, the script below assembles them into a complete Grafana dashboard JSON — cascading template variables, correct histogram_quantile calls, deploy annotations, and per-panel trace links — then writes the result to disk where Terraform or grafana-operator will pick it up.

Code snippetpython
1# tools/render_api_dashboard.py
2import json, pathlib
3
4DS = "$datasource"
5
6def panel(title, expr, unit="short", runbook="https://runbooks.example.com/api"):
7    return {
8        "title": title,
9        "type": "timeseries",
10        "datasource": DS,
11        "targets": [{"expr": expr, "datasource": DS}],
12        "fieldConfig": {"defaults": {"unit": unit}},
13        "description": f"Runbook: {runbook}",
14        "links": [{
15            "title": "View traces",
16            "url": ('/explore?orgId=1&left={"datasource":"tempo",'
17                    '"queries":[{"query":"{service=\\"$service\\",'
18                    'http.route=\\"$route\\"}"}],'
19                    '"range":{"from":"$__from","to":"$__to"}}'),
20        }],
21    }
22
23P95 = ('histogram_quantile(0.95, sum by (le, route) ('
24       'rate(http_request_duration_seconds_bucket'
25       '{service="$service",route=~"$route"}[5m])))')
26ERR = ('100 * sum(rate(http_requests_total'
27       '{service="$service",status=~"5.."}[5m])) / '
28       'sum(rate(http_requests_total{service="$service"}[5m]))')
29RPS = 'sum by (route) (rate(http_requests_total{service="$service"}[1m]))'
30LAG = ('histogram_quantile(0.99, rate('
31       'python_asyncio_event_loop_lag_seconds_bucket'
32       '{service="$service"}[5m]))')
33
34dashboard = {
35    "title": "API — Golden Signals",
36    "templating": {"list": [
37        {"name": "datasource", "type": "datasource", "query": "prometheus"},
38        {"name": "namespace",  "type": "query",
39         "query": "label_values(namespace)"},
40        {"name": "service",    "type": "query",
41         "query": 'label_values(http_requests_total'
42                  '{namespace="$namespace"}, service)'},
43        {"name": "route",      "type": "query",
44         "query": 'label_values(http_requests_total'
45                  '{service="$service"}, route)'},
46    ]},
47    "annotations": {"list": [{
48        "name": "Deploys",
49        "datasource": DS,
50        "expr": ('changes(kube_deployment_status_observed_generation'
51                 '{namespace="$namespace",deployment="$service"}[1m]) > 0'),
52        "iconColor": "rgba(0, 211, 255, 1)",
53        "titleFormat": "Deploy of $service",
54    }]},
55    "panels": [
56        panel("P95 latency by route", P95, "s"),
57        panel("Error rate %",         ERR, "percent"),
58        panel("Traffic (req/s)",      RPS, "reqps"),
59        panel("Event-loop lag P99",   LAG, "s"),
60    ],
61}
62
63pathlib.Path("dashboards/api-golden.json").write_text(
64    json.dumps(dashboard, indent=2)
65)

The four template variables cascade in the order they appear: $namespace scopes the cluster, $service filters every panel to one API deployment, and $route lets you pivot from a service-wide error rate spike to the single endpoint responsible. The P95 expression groups by both le and route — omitting le collapses the histogram buckets into meaningless averages, which is the mistake the histogram_quantile entry in Key Terminology warns against. The annotations block injects a vertical deploy line by detecting changes in kube_deployment_status_observed_generation, so an on-call engineer can tell at a glance whether a rollout caused the spike. Every panel carries a links entry that constructs a Tempo deep-link scoped to the same $service, $route, and dashboard time window, completing the metrics-to-traces path described in the three-tier layout. Storing this file in git and rendering it with python tools/render_api_dashboard.py is the dashboard-as-code practice that keeps panel definitions and alert thresholds in sync across environments.

You'll know it works when python tools/render_api_dashboard.py writes dashboards/api-golden.json, importing that file into Grafana renders four tier-one panels with $service and $route dropdowns populated at the top, a vertical annotation line appears on each panel the next time the deployment rolls, and clicking any panel opens Tempo pre-filtered to the same service, route, and time window.

Do's and Don'ts

Building on the patterns above, here are the highest-leverage habits to lock in — and the traps that quietly turn a dashboard into wallpaper.

Do's

1. ✓Do put golden signals on the top row at full width — so the on-call engineer answers "is anything wrong?" before scrolling, in line with Google SRE practice.
2. ✓Do store dashboards as JSON in git and render via Grafonnet or Terraform — UI edits drift silently and have no review trail.
3. ✓Do embed the runbook URL and alert thresholds on every panel — the dashboard then doubles as the incident playbook and the visual severity matches what PagerDuty paged on.

Don'ts

1. ✗Don't average _sum / _count to estimate latency — always use histogram_quantile; averages mask the tail where users actually feel pain.
2. ✗Don't template on unbounded labels like $user_id — high-cardinality variables crash Prometheus and break every dashboard at once.
3. ✗Don't widen rate windows past [5m] on tier-one panels — a 30-minute rate looks calm during a 2-minute outage and the on-call sees nothing.