Implement dashboard-as-code with Grafana provisioning for version-controlled dashboards

Introduction

Teams that manage GenAI platforms often build Grafana dashboards manually through the UI — each panel configured by hand, with no record of what changed or why. When a dashboard breaks or needs replication across environments, there is no source of truth to restore from. Dashboard-as-code solves this by treating every panel, query, and threshold as a versioned artifact stored in Git, provisioned automatically through Kubernetes ConfigMaps and Grafana's sidecar-based loader. By the end of this lesson, you'll be able to export Grafana dashboards to normalized JSON, store them in Git, and provision them through Kubernetes so that every dashboard change is tracked, reviewable, and automatically deployed.

Key Terminology

• Dashboard-as-code — a practice where every Grafana dashboard is stored as a versioned JSON file in Git rather than maintained as mutable state inside Grafana's internal database, making changes reviewable, auditable, and reproducible across environments.
• Dashboard normalization — the process of stripping runtime-only fields such as id, version, and iteration from exported dashboard JSON before committing to Git, implemented in the _normalize method of DashboardExporter, so that Git diffs reflect only meaningful changes to queries, panels, and thresholds.
• Grafana provisioning sidecar — a container that runs alongside Grafana in Kubernetes, watches for ConfigMap resources carrying a specific label, and copies matching JSON files into Grafana's provisioning directory so dashboards are loaded automatically without a pod restart or manual import.
• ConfigMap (dashboard carrier) — a Kubernetes ConfigMap resource that embeds normalized dashboard JSON as a data key and carries the grafana_dashboard: "1" label that the provisioning sidecar uses as its discovery signal.
• GrafanaExportConfig — a Pydantic model that validates the Grafana API key length, output directory path, and normalization flags before any network calls are made, serving as the typed configuration contract for the DashboardExporter.

Concepts

Why Mutable Dashboards Break Operations

Grafana's default workflow stores every dashboard as live state in its internal SQLite or Postgres database. This is fast for exploration, but it creates a fundamental ops problem: there is no history of what changed, no review step before a change goes live, and no authoritative source to restore from when a dashboard breaks or needs to be reproduced in a staging environment. Two engineers can each edit the same dashboard from different browser tabs, and the last write wins with no conflict record.

Dashboard-as-code inverts this model. The JSON file in Git becomes the source of truth; the running Grafana instance is a derived artifact. A dashboard change follows the same path as any code change — pull request, review, merge — and the deployment path is identical to any other Kubernetes resource update.

The Two-Step Pipeline: Export Then Provision

The implementation separates two concerns that are easy to conflate. The DashboardExporter class handles the export direction: it reads the current state of Grafana via its HTTP API (/api/search to enumerate dashboards, /api/dashboards/uid/{uid} to fetch each one), strips non-semantic fields in _normalize, and writes the resulting JSON files to a local output directory keyed by UID. This step produces the Git-committed artifacts.

Provisioning runs in the opposite direction: the normalized JSON files are embedded in a Kubernetes ConfigMap and the grafana_dashboard: "1" label tells the sidecar to treat that ConfigMap as a dashboard source. When Argo CD syncs the ConfigMap after a Git merge, the sidecar detects the change and reloads the dashboard in Grafana — no kubectl exec, no manual import, no pod restart required (see Code Walkthrough).

Normalization as Git Hygiene

Grafana's export API returns dashboards with several auto-managed fields: a database-assigned id integer, an auto-incrementing version counter, and an iteration timestamp updated on every save. None of these fields describe what the dashboard does — they describe Grafana's internal bookkeeping. If they are left in the committed JSON, every routine save inside Grafana's UI produces a Git diff consisting entirely of counter increments, making code review meaningless and blame history unreadable.

The _normalize method in DashboardExporter removes these fields unconditionally when the corresponding config flags are set. The result is that a commit diff shows only changes to panel queries, threshold values, layout geometry, or tag lists — the information a reviewer actually needs to evaluate whether the change is correct. This is why normalization is not optional polish; it is the prerequisite that makes dashboard-as-code practically useful rather than just theoretically sound.

Code Walkthrough

Now that you understand how Grafana's provisioning system separates dashboard definitions from runtime state, the implementation follows two concrete steps: exporting dashboards to normalized JSON and wiring them into Kubernetes as ConfigMaps.

The DashboardExporter class connects to Grafana's HTTP API, fetches every dashboard by UID, and strips the runtime fields — auto-incrementing IDs, version counters, and iteration timestamps — that would otherwise pollute Git diffs with meaningless noise. The _normalize method performs this stripping before the dashboard JSON is written to disk.

Code snippetpython
1import json
2import httpx
3from pathlib import Path
4from pydantic import BaseModel, Field
5
6class GrafanaExportConfig(BaseModel):
7    grafana_url: str = "http://localhost:3000"
8    api_key: str = Field(..., min_length=10)
9    output_dir: Path = Path("dashboards/")
10    strip_ids: bool = True
11    strip_version: bool = True
12
13class DashboardExporter:
14    def __init__(self, config: GrafanaExportConfig):
15        self.config = config
16        self.client = httpx.Client(
17            base_url=config.grafana_url,
18            headers={"Authorization": f"Bearer {config.api_key}"},
19            timeout=30.0,
20        )
21
22    def list_dashboards(self) -> list[dict]:
23        response = self.client.get("/api/search?type=dash-db")
24        response.raise_for_status()
25        return response.json()
26
27    def export_dashboard(self, uid: str) -> dict:
28        response = self.client.get(f"/api/dashboards/uid/{uid}")
29        response.raise_for_status()
30        raw = response.json()
31        dashboard = raw.get("dashboard", raw)
32        return self._normalize(dashboard)
33
34    def _normalize(self, dashboard: dict) -> dict:
35        if self.config.strip_ids:
36            dashboard.pop("id", None)
37            for panel in dashboard.get("panels", []):
38                panel.pop("id", None)
39        if self.config.strip_version:
40            dashboard.pop("version", None)
41        dashboard.pop("iteration", None)
42        return dashboard
43
44    def export_all(self) -> dict[str, Path]:
45        self.config.output_dir.mkdir(parents=True, exist_ok=True)
46        exported = {}
47        for item in self.list_dashboards():
48            uid = item["uid"]
49            dashboard = self.export_dashboard(uid)
50            path = self.config.output_dir / f"{uid}.json"
51            with open(path, "w") as f:
52                json.dump(dashboard, f, indent=2)
53            exported[uid] = path
54        return exported

GrafanaExportConfig uses Pydantic to validate the API key length and the output directory path before any network calls are made. export_all creates the output directory when it is missing, then iterates every dashboard returned by Grafana's search endpoint and writes each normalized dashboard to a JSON file named by its UID. Because _normalize has already removed the auto-incrementing id and version fields, committing these files to Git produces diffs that reflect only meaningful changes to panel queries, thresholds, and layout.

With the JSON files committed to Git, the next step is provisioning them into Grafana through Kubernetes. Grafana's sidecar container watches for ConfigMap resources labeled grafana_dashboard: "1" and copies matching JSON files into Grafana's provisioning directory, making a dashboard update indistinguishable from any other Kubernetes resource sync that Argo CD manages.

Code snippetyaml
1# grafana-dashboard-provisioning.yaml
2apiVersion: v1
3kind: ConfigMap
4metadata:
5  name: grafana-dashboards
6  namespace: monitoring
7  labels:
8    grafana_dashboard: "1"
9data:
10  genai-operations.json: |
11    {
12      "title": "GenAI Operations",
13      "uid": "genai-ops",
14      "panels": [],
15      "schemaVersion": 36,
16      "tags": ["genai", "operations"]
17    }

The grafana_dashboard: "1" label is the discovery signal the sidecar container uses. When Argo CD syncs this ConfigMap after a Git merge, the sidecar detects the change and reloads the dashboard in Grafana without requiring a pod restart or a manual import step.

Confirm that the full pipeline works end-to-end: run export_all() against a live Grafana instance, commit the resulting JSON files to Git, apply the ConfigMap to your cluster, and verify that the dashboard appears in Grafana's UI under the correct folder with no manual intervention required.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do call _normalize before writing any dashboard JSON to disk — stripping id, version, and iteration from every panel and the top-level object ensures Git diffs reflect only meaningful changes to queries, thresholds, and layout, not Grafana's auto-incrementing counters.
2. ✓Do apply the grafana_dashboard: "1" label to every dashboard ConfigMap — this label is the exact discovery signal the Grafana sidecar container uses to identify which ConfigMaps to copy into the provisioning directory; omitting it means the sidecar ignores the resource entirely and dashboards never load.
3. ✓Do name exported dashboard files by UID ({uid}.json) — Grafana uses the uid field as the stable deduplication key across reloads; using an arbitrary filename while leaving the uid field intact lets Argo CD and the sidecar reload the correct dashboard in place without creating duplicates.

Don'ts

1. ✗Don't commit raw Grafana API export payloads without running _normalize — Grafana's /api/dashboards/uid/{uid} response includes id, version, and iteration fields that increment on every UI save, flooding pull request diffs with noise that hides real panel or query changes.
2. ✗Don't skip Pydantic validation in GrafanaExportConfig by constructing the client with a hard-coded or empty api_key — GrafanaExportConfig enforces min_length=10 precisely to catch placeholder tokens before any HTTP calls are made; bypassing it lets a misconfigured key reach the Grafana API and return 401 errors mid-export with no dashboards written.
3. ✗Don't manually re-import a provisioned dashboard through the Grafana UI after the ConfigMap pipeline is established — the sidecar owns the provisioned copy and will overwrite UI edits on the next Argo CD sync, so hand-edits are silently discarded and never reach Git.