Deploy cost dashboards with Grafana

Introduction

Engineers often have LLM cost data sitting in a PostgreSQL events table and a Prometheus scrape endpoint with no unified view that shows which teams are over budget or which models are driving spend. Without a live dashboard, operators resort to ad-hoc SQL queries and miss budget breaches until the billing cycle closes. This lesson shows you how to deploy a Grafana dashboard that pulls daily cost trends from PostgreSQL and real-time utilization metrics from Prometheus, filtered by department and team variables. By the end, you'll be able to define a dashboard JSON with variable templates and budget thresholds, then provision it into Kubernetes as a ConfigMap so Grafana loads the view automatically.

Key Terminology

• Grafana provisioning sidecar — a Kubernetes sidecar container that watches a mounted volume in the monitoring namespace and automatically loads any .json file it finds, so a ConfigMap-delivered dashboard appears in Grafana's UI without a manual import step.
• Variable template — a Grafana dashboard object of type: query that populates a dropdown filter by executing a SQL query against a data source; the lesson defines three — department, team, and model — that cascade so selecting a department automatically narrows the team list and filters all panel queries.
• Dual data source dashboard — a Grafana pattern that routes different panels to different backends; here the timeseries panel targets PostgreSQL for historical cost_events aggregations while the gauge panel targets Prometheus for the live platform_budget_utilization_ratio metric.
• Budget utilization threshold — a fieldConfig.defaults.thresholds.steps array on the gauge panel that maps ratio values to colors: green below 0.7, yellow from 0.7, and red from 0.9, giving operators an at-a-glance signal of budget pressure without reading raw numbers.
• Kubernetes ConfigMap — a v1 Kubernetes resource whose data field carries the Grafana dashboard JSON under a .json-suffixed key; applied to the monitoring namespace, it delivers the dashboard to Grafana's provisioning sidecar without any manual cluster login or UI action.
• Dashboard serialization — the step of calling json.dumps(configmap, indent=2) to convert the Python dict produced by provision_as_configmap into the JSON text that kubectl apply -f can consume, completing the pipeline from Python function to cluster resource.

Concepts

Why Two Data Sources Instead of One

A cost dashboard needs to answer two fundamentally different questions: how was money spent over time, and how close is a team to its budget right now? PostgreSQL and Prometheus serve these roles without overlap.

PostgreSQL holds the cost_events table — an accumulating record of every model invocation with its total_cost, tenant_id, and timestamp. Aggregating with date_trunc('day', timestamp) and GROUP BY produces the stable daily trend that engineering leads and finance teams actually review: what was spent, by whom, on which days. Prometheus, by contrast, continuously scrapes platform_budget_utilization_ratio as a live ratio. It answers "is this team burning through budget faster than expected right now?" rather than "what did they spend last Tuesday?" Its short retention window makes it unsuitable for month-to-date trend queries, but it is exactly right for a real-time gauge.

Routing the timeseries panel to PostgreSQL and the gauge panel to Prometheus (see Code Walkthrough) avoids the failure modes of each system used alone: PostgreSQL cannot produce a continuously updated gauge without expensive streaming infrastructure, and Prometheus cannot reconstruct a billing-cycle trend from a 15-day scrape window.

Variable Templates as Cascading Drill-Down

Dashboard variable templates let a single JSON definition serve every team and department without duplicating panels. The three templates — department, team, and model — are all declared as type: query, meaning Grafana re-executes the underlying SQL each time an operator changes a selection. The team query filters on '$department', creating a cascade: choosing "Platform Engineering" in the department dropdown automatically restricts the team list to teams within that department. The same $department token appears verbatim inside the timeseries panel's rawSql subquery, so chart data narrows to match without any additional panel configuration.

This pattern eliminates the N-panels-per-team antipattern — a dashboard with one panel hardcoded per team that becomes unmaintainable past a dozen teams. With variable templates, operators drill from an org-wide view down to a single team's daily spend in two dropdown clicks, and the dashboard JSON stays a fixed size regardless of how many teams exist.

ConfigMap Provisioning and the Sidecar Pattern

Grafana supports file-based dashboard provisioning: any .json file placed in a watched directory — either at startup or via a volume mount into a running pod — is loaded automatically. In Kubernetes, a ConfigMap is the standard mechanism for injecting configuration files into a pod's filesystem. The provision_as_configmap function (see Code Walkthrough) constructs the required apiVersion: v1 / kind: ConfigMap manifest, places the dashboard JSON under a .json-keyed data entry, and targets the monitoring namespace where Grafana's sidecar watches for new files.

Once kubectl apply -f processes the serialized output, the sidecar detects the new ConfigMap on its next refresh cycle and registers the dashboard — no Grafana UI login, no manual import, no API call required. This matters operationally: dashboard definitions become code changes tracked in version control and deployed through the same pipeline as any other Kubernetes manifest, rather than configuration that lives only inside Grafana's internal database.

The Serialization Step: From Python Dict to kubectl Manifest

The final step converts a nested Python dict into text that kubectl apply accepts. json.dumps(configmap, indent=2) produces indented JSON, which kubectl treats identically to YAML. The nesting structure is deliberate: the outer dict carries the Kubernetes object fields (apiVersion, kind, metadata, data), while data["{name}.json"] holds the Grafana dashboard JSON as a plain string value — produced by an inner json.dumps(dashboard["dashboard"]) call without indentation, since Grafana's sidecar parses it programmatically. Keeping these two serialization passes separate is important: a single json.dumps of the entire structure would embed Kubernetes metadata inside the Grafana JSON object, which the sidecar would reject, while double-encoding data as a nested JSON-in-JSON string produces a ConfigMap that kubectl itself will refuse.

Code Walkthrough

Now that you understand how PostgreSQL cost queries, Prometheus real-time metrics, and Kubernetes ConfigMap provisioning fit together as a unified monitoring stack, the implementation has two moving parts: the dashboard JSON definition and the ConfigMap that delivers it to Grafana.

The function below defines three variable templates — department, team, and model — so operators can drill from organization-wide totals down to a single team's spending. Two panel types cover the core views: a timeseries panel queries PostgreSQL for daily cost trends filtered by the selected department, and a gauge panel reads the Prometheus metric platform_budget_utilization_ratio and applies green/yellow/red thresholds so budget pressure is visible at a glance. Budget threshold alerts from Alertmanager annotate the time-series panel automatically once the Grafana data source connection is live.

Code snippetpython
1import json
2
3def generate_cost_overview_dashboard() -> dict:
4    return {
5        "dashboard": {
6            "title": "LLM Cost Overview",
7            "tags": ["cost", "llm", "platform"],
8            "templating": {
9                "list": [
10                    {
11                        "name": "department",
12                        "type": "query",
13                        "query": "SELECT DISTINCT department FROM tenants",
14                    },
15                    {
16                        "name": "team",
17                        "type": "query",
18                        "query": "SELECT team_id FROM tenants WHERE department='$department'",
19                    },
20                    {
21                        "name": "model",
22                        "type": "query",
23                        "query": "SELECT DISTINCT model_id FROM cost_events",
24                    },
25                ]
26            },
27            "panels": [
28                {
29                    "title": "Daily Cost Trend",
30                    "type": "timeseries",
31                    "datasource": "PostgreSQL",
32                    "targets": [{
33                        "rawSql": (
34                            "SELECT date_trunc('day', timestamp) AS time, "
35                            "SUM(total_cost) AS cost "
36                            "FROM cost_events "
37                            "WHERE tenant_id IN ("
38                            "  SELECT team_id FROM tenants "
39                            "  WHERE department='$department') "
40                            "GROUP BY 1 ORDER BY 1"
41                        ),
42                    }],
43                    "fieldConfig": {"defaults": {"unit": "currencyUSD"}},
44                },
45                {
46                    "title": "Budget Utilization",
47                    "type": "gauge",
48                    "datasource": "Prometheus",
49                    "targets": [{"expr": 'platform_budget_utilization_ratio{department="$department"}'}],
50                    "fieldConfig": {
51                        "defaults": {
52                            "thresholds": {
53                                "steps": [
54                                    {"value": 0,   "color": "green"},
55                                    {"value": 0.7, "color": "yellow"},
56                                    {"value": 0.9, "color": "red"},
57                                ]
58                            }
59                        }
60                    },
61                },
62            ],
63        }
64    }

To deploy the dashboard, wrap it in a Kubernetes ConfigMap. Grafana's provisioning sidecar watches the monitoring namespace and loads any .json file it finds in the mounted volume, so the dashboard appears in the Grafana UI without a manual import step.

Code snippetpython
1def provision_as_configmap(dashboard: dict, name: str = "cost-overview") -> dict:
2    return {
3        "apiVersion": "v1",
4        "kind": "ConfigMap",
5        "metadata": {
6            "name": f"grafana-{name}",
7            "namespace": "monitoring",
8        },
9        "data": {f"{name}.json": json.dumps(dashboard["dashboard"])},
10    }
11
12# Serialize the ConfigMap for kubectl apply
13configmap = provision_as_configmap(generate_cost_overview_dashboard())
14print(json.dumps(configmap, indent=2))

After running kubectl apply -f with the printed output, Grafana's provisioning sidecar picks up the file on its next refresh cycle and the dashboard becomes available under the cost tag. Confirm that the Daily Cost Trend panel returns data rows and the Budget Utilization gauge shows a non-zero value for at least one department before moving on.

Do's and Don'ts

Having just walked through the dashboard JSON, the ConfigMap wrapper, and the provisioning sidecar pipeline, these rules keep that pipeline from silently failing once it lands in the monitoring namespace.

Do's

1. ✓Do serialize only the inner dashboard["dashboard"] dict into the ConfigMap's data field — provision_as_configmap calls json.dumps(dashboard["dashboard"]), not json.dumps(dashboard), because Grafana's provisioning sidecar expects the raw dashboard object as the .json file content; including the outer {"dashboard": …} wrapper produces a file the sidecar silently rejects.
2. ✓Do place the ConfigMap in the monitoring namespace with a .json-suffixed data key — Grafana's provisioning sidecar watches the mounted volume in that namespace and only auto-loads files whose keys end in .json; using the wrong namespace or a different extension means the dashboard never appears in the UI without a manual import step.
3. ✓Do thread the $department variable through both the PostgreSQL subquery and the Prometheus label selector — the timeseries panel filters cost_events via WHERE department='$department' through the tenants subquery, and the gauge panel uses {department="$department"} on platform_budget_utilization_ratio; both must reference the same variable so the two panels stay in sync when an operator changes the department filter.

Don'ts

1. ✗Don't pass the full generate_cost_overview_dashboard() return value directly to json.dumps in the ConfigMap — the function returns {"dashboard": {…}} as a Python convention, but only the inner dict is the valid Grafana provisioning payload; wrapping the wrapper produces a top-level "dashboard" key inside the .json file that the sidecar cannot parse as a dashboard definition.
2. ✗Don't omit the team variable template that cascades off $department — the query SELECT team_id FROM tenants WHERE department='$department' is what lets operators drill from organization-wide cost totals down to per-team spend; without it the dashboard can only filter at department granularity and the chargeback reporting use case collapses to a single aggregation level.
3. ✗Don't skip confirming that the platform_budget_utilization_ratio gauge returns a non-zero value after kubectl apply — a gauge stuck at zero means the Prometheus data source is misconfigured or the department label on the metric doesn't match the variable value, and the green/yellow/red threshold steps (0.7 and 0.9) will never fire regardless of actual budget pressure.