Deploy Grafana OnCall for On-Call Schedules, Escalation Policies, and Incident Lifecycle

Introduction

When you operate a GenAI alert system in production, Alertmanager alone cannot tell you who is currently on-call, whether a critical alert was acknowledged, or how to escalate when no one responds. You need a dedicated on-call management layer that bridges alert routing to human incident workflows. By the end of this lesson, you'll be able to deploy Grafana OnCall using its Helm chart, configure on-call rotation schedules, define escalation chains that page the right responders when alerts go unacknowledged, and understand how the incident lifecycle flows from alert ingestion through resolution.

Key Terminology

• Grafana OnCall — A dedicated on-call management layer deployed via Helm chart that bridges Alertmanager's routing engine to human incident workflows, providing rotation schedules, escalation chains, and acknowledgment tracking that Alertmanager alone cannot supply.
• OnCall Engine — The central HTTP service (exposed at oncall_base_url) that receives alert payloads from Alertmanager via a webhook receiver and drives the incident lifecycle; configured in OnCallConfig via engine_replicas and BASE_URL env vars.
• Escalation Chain — An ordered sequence of notification steps that OnCall executes when an alert goes unacknowledged, automatically paging the next responder tier after a configurable timeout window.
• Rotation Schedule — A time-based assignment of on-call duty to team members, built by OnCallScheduleManager._build_shifts() as a series of recurrent_event shifts spaced by rotation_interval_days and anchored to a weekly handoff_hour and handoff_day.
• Alertmanager Receiver (webhook) — An Alertmanager configuration stanza, generated by OnCallDeployment.generate_alertmanager_receiver(), that directs webhook traffic to the OnCall engine's /integrations/v1/alertmanager/ ingestion endpoint using bearer token authentication.
• External Redis — A Redis instance referenced by externalRedis.host in the Helm values (with the bundled Redis disabled via redis.enabled: False) that persists OnCall's Celery task queue state outside the pod lifecycle, preventing task loss on pod restarts.

Concepts

Why Alertmanager Alone Is Not Enough

Alertmanager is a routing and deduplication engine: it groups related alerts, suppresses duplicates, and dispatches notifications to configured receivers. What it cannot do is answer the question who is on-call right now? or enforce that a critical page was actually seen and acknowledged. When a PagerDuty-style workflow is required — rotating weekly duty, escalation after five minutes of silence, and a clear incident lifecycle from "firing" to "resolved" — you need a second system that speaks Alertmanager's webhook protocol but adds the human-coordination layer on top.

Grafana OnCall fills exactly that gap. It exposes an HTTP endpoint (/integrations/v1/alertmanager/) that Alertmanager treats as an ordinary webhook receiver. Once an alert arrives, OnCall takes over: it looks up the current on-call person from a rotation schedule, pages them through their preferred channel (Slack, SMS, voice), and—if the alert goes unacknowledged—walks an escalation chain until someone responds. The Code Walkthrough materializes both sides of this connection: the Helm values that deploy the engine, and the Alertmanager receiver stanza that points at it (see Code Walkthrough).

The Deployment Artifact Pair

Deploying OnCall requires two coordinated artifacts, and keeping them generated from the same OnCallConfig dataclass prevents them from drifting apart:

1. Helm values — tell the Kubernetes cluster how many engine and Celery worker replicas to run, where Grafana lives, and which Redis instance to use for the task queue. Disabling the bundled Redis (redis.enabled: False) and pointing at an external instance is a deliberate production hardening choice: Celery jobs that track escalation state survive pod restarts because their queue lives outside the pod's lifecycle.

2. Alertmanager receiver stanza — registers grafana-oncall as a webhook receiver in Alertmanager's routing tree. The bearer token in http_config ensures only authorized Alertmanager instances can push alerts into OnCall, preventing spoofed alert injections.

Because both artifacts derive from the same OnCallConfig, changing oncall_base_url in one place propagates correctly to both the engine's BASE_URL environment variable and the webhook URL that Alertmanager calls.

Rotation Schedules and Shift Construction

A rotation schedule is a set of time-bounded, recurrently firing assignments. OnCallScheduleManager._build_shifts() constructs these shifts by iterating over the RotationConfig.team_members list and offsetting each member's start time by one rotation_interval_days multiple. The result is a chain: member 0 is on-call from now, member 1 takes over one interval later, member 2 one interval after that, and so on — all anchored to the same handoff_hour so duty changes happen at a predictable time.

Once create_schedule() posts the shift list to /api/v1/schedules/, the OnCall engine continuously evaluates the current wall-clock time against each shift's start and duration to determine who is on-call at any moment — making the schedule the live source of truth for escalation decisions.

Code Walkthrough

Now that you understand how Grafana OnCall bridges Alertmanager's routing engine to human incident workflows, the following examples show how to generate the deployment artifacts that wire these two systems together.

The OnCallConfig dataclass collects every configurable parameter — replica counts, service URLs, and the integration endpoints linking the OnCall engine to Alertmanager and Grafana. OnCallDeployment consumes that config to produce two outputs: Helm values that drive helm install, and an Alertmanager receiver stanza that directs webhook traffic to the OnCall engine's alert ingestion endpoint.

Code snippetpython
1from dataclasses import dataclass
2import yaml
3
4@dataclass
5class OnCallConfig:
6    namespace: str = "monitoring"
7    release_name: str = "grafana-oncall"
8    engine_replicas: int = 2
9    celery_replicas: int = 2
10    redis_url: str = "redis://redis:6379/0"
11    grafana_url: str = "http://grafana:3000"
12    alertmanager_url: str = "http://alertmanager:9093"
13    oncall_base_url: str = "http://grafana-oncall-engine:8080"
14
15class OnCallDeployment:
16    def __init__(self, config: OnCallConfig):
17        self.config = config
18
19    def generate_helm_values(self) -> dict:
20        return {
21            "engine": {
22                "replicaCount": self.config.engine_replicas,
23                "env": [
24                    {"name": "BASE_URL", "value": self.config.oncall_base_url},
25                    {"name": "GRAFANA_API_URL", "value": self.config.grafana_url},
26                ],
27            },
28            "celery": {"replicaCount": self.config.celery_replicas},
29            "redis": {"enabled": False},
30            "externalRedis": {"host": self.config.redis_url},
31            "ingress": {"enabled": False},
32        }
33
34    def generate_alertmanager_receiver(self) -> dict:
35        webhook_url = (
36            f"{self.config.oncall_base_url}"
37            "/integrations/v1/alertmanager/"
38        )
39        return {
40            "name": "grafana-oncall",
41            "webhook_configs": [
42                {
43                    "url": webhook_url,
44                    "send_resolved": True,
45                    "http_config": {
46                        "bearer_token_file": "/etc/alertmanager/secrets/oncall-api-token",
47                    },
48                }
49            ],
50        }

The Helm values disable the bundled Redis and reference an external instance — a production pattern that prevents task-queue data loss if the OnCall pod restarts. The Alertmanager receiver uses bearer token authentication so only authorized Alertmanager instances can push alerts into OnCall.

Once the engine is running, on-call schedules connect the alert stream to real people. OnCallScheduleManager calls the Grafana OnCall REST API to create rotation-based schedules. _build_shifts translates a RotationConfig into a list of recurrent shift objects spaced by the configured interval, cycling through team members starting from the next scheduled handoff window.

Code snippetpython
1from dataclasses import dataclass
2from datetime import datetime, timedelta
3from typing import List
4import httpx
5
6@dataclass
7class TeamMember:
8    name: str
9    email: str
10    slack_id: str
11    timezone: str = "America/New_York"
12
13@dataclass
14class RotationConfig:
15    name: str
16    team_members: List[TeamMember]
17    rotation_interval_days: int = 7
18    handoff_hour: int = 9
19    handoff_day: str = "monday"
20
21class OnCallScheduleManager:
22    def __init__(self, oncall_base_url: str, api_token: str):
23        self.base_url = oncall_base_url
24        self.headers = {
25            "Authorization": f"Token {api_token}",
26            "Content-Type": "application/json",
27        }
28
29    def create_schedule(self, rotation: RotationConfig) -> dict:
30        shifts = self._build_shifts(rotation)
31        payload = {
32            "name": rotation.name,
33            "type": "web",
34            "time_zone": rotation.team_members[0].timezone,
35            "shifts": shifts,
36        }
37        with httpx.Client() as client:
38            response = client.post(
39                f"{self.base_url}/api/v1/schedules/",
40                headers=self.headers,
41                json=payload,
42                timeout=10.0,
43            )
44            response.raise_for_status()
45            return response.json()
46
47    def _build_shifts(self, rotation: RotationConfig) -> list:
48        shifts = []
49        start = datetime.now().replace(
50            hour=rotation.handoff_hour, minute=0, second=0, microsecond=0
51        )
52        interval = timedelta(days=rotation.rotation_interval_days)
53        for i, member in enumerate(rotation.team_members):
54            shifts.append({
55                "name": f"{rotation.name}-shift-{i + 1}",
56                "type": "recurrent_event",
57                "start": (start + interval * i).isoformat(),
58                "duration": int(interval.total_seconds()),
59                "users": [member.email],
60                "frequency": "weekly",
61            })
62        return shifts

Confirm that generate_helm_values() returns a dict with an "engine" key whose "replicaCount" matches OnCallConfig.engine_replicas, and that create_schedule() returns a response body containing a "name" field equal to the rotation name you passed in — both verify that the Helm configuration and OnCall API integration are correctly wired.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do disable the Helm chart's bundled Redis by setting "redis": {"enabled": False} and supplying externalRedis.host in generate_helm_values() — the OnCall engine's Celery workers write in-flight escalation tasks to Redis, and coupling that queue to the OnCall pod means any pod restart or upgrade silently drops pending pages before they fire.
2. ✓Do set "send_resolved": True in the Alertmanager receiver's webhook_configs stanza produced by generate_alertmanager_receiver() — without it, OnCall never receives Alertmanager's resolution notification, leaving incidents stuck open and escalation chains continuing to page responders long after the underlying alert condition has cleared.
3. ✓Do authenticate the Alertmanager → OnCall webhook using bearer_token_file pointing to a mounted Kubernetes secret in http_config — this ensures only authorized Alertmanager instances can push payloads into the OnCall engine's /integrations/v1/alertmanager/ ingestion endpoint, and lets you rotate credentials without editing the receiver stanza.

Don'ts

1. ✗Don't point the Alertmanager receiver's url at the bare oncall_base_url — generate_alertmanager_receiver() appends /integrations/v1/alertmanager/ to form the actual ingestion path; omitting that suffix causes every Alertmanager webhook delivery to return a 404, and OnCall receives no alerts while Alertmanager logs silent failures.
2. ✗Don't run the OnCall Helm chart with default bundled Redis (redis.enabled: true) in production — task-queue state is co-located inside the same release as the engine pod, so a rolling upgrade or crash-restart drops queued escalation events and creates a missed-page window that no retry logic in OnCallDeployment can recover.
3. ✗Don't build on-call shifts without converting timedelta.total_seconds() to int before posting to /api/v1/schedules/ — the OnCall REST API rejects float-typed duration values, and create_schedule() in OnCallScheduleManager must call int(interval.total_seconds()) so the recurrent shift objects pass schema validation and the rotation hands off at the correct weekly boundary.