Build secret sync monitoring and alerting for rotation compliance

Introduction

When External Secrets Operator synchronizes provider keys from GCP Secret Manager into Kubernetes, sync failures happen silently — a misconfigured SecretStore, a revoked IAM binding, or an expired service account key stalls synchronization while your workloads keep running on stale credentials until inference failures surface at the worst possible moment. Production GenAI systems relying on OpenAI, Anthropic, and GCP keys cannot tolerate this blind spot.

By the end of this lesson, you will build a complete monitoring pipeline — a sync-status collector, a rotation compliance evaluator, and an environment-aware alert dispatcher — that detects drift within minutes, enforces per-provider rotation windows, and routes violations to PagerDuty or Slack before stale keys cause failures.

Key terminology

• Sync drift: The time delta between the last successful ExternalSecret sync and the current time, measured in hours. Drift exceeding the rotation policy threshold indicates a compliance violation.

• Rotation compliance window: The maximum allowable age of a synchronized secret before it must be rotated, typically 90 days (2160 hours) for provider API keys and 30 days (720 hours) for service account credentials.

• Namespace-derived severity: The rule that an alert's severity is computed from the secret's namespace rather than per-secret configuration — prod violations escalate as critical and route to PagerDuty, while staging violations are warning and route to Slack — keeping the policy consistent and auditable.

• Environment isolation: The practice of routing alerts to environment-specific channels and escalation chains so that staging noise never masks production incidents and development alerts never page on-call engineers.

• SecretSynced condition: The Kubernetes status condition that External Secrets Operator writes on each ExternalSecret resource, carrying a status of True or False, a lastTransitionTime timestamp, and an optional error message when sync fails.

Concepts

The silent failure problem in secret synchronization

When External Secrets Operator fails to synchronize a key — because a SecretStore is misconfigured, an IAM binding has been revoked, or a service account key has expired — it writes the failure onto the ExternalSecret object's status conditions and stops there. Your workloads continue running on whatever credentials were last written into the Kubernetes Secret, and nothing in the default Kubernetes control plane escalates that condition to a human. The gap between "ESO stopped syncing" and "inference requests start failing" can be hours or days, depending on how long cached credentials remain valid.

This is the detection gap the monitoring pipeline closes. Rather than waiting for downstream failures to surface the problem, the pipeline polls the SecretSynced condition directly — the same condition ESO writes — and converts a silent Kubernetes object-level status into an active alert within one polling cycle. The SyncStatusCollector is purpose-built for this: it speaks the Custom Objects API to enumerate every ExternalSecret across your namespaces, reads the operator-written condition, and materializes the result into structured SecretSyncRecord instances that downstream stages can evaluate without touching Kubernetes again.

Separation of concerns: collector, evaluator, and dispatcher

The pipeline is deliberately split into three independent stages rather than a single function that fetches and alerts. This separation reflects a real operational requirement: the logic for detecting a sync failure, deciding whether it constitutes a violation, and routing that violation are all likely to change at different rates and for different reasons.

The SyncStatusCollector owns only the Kubernetes-query boundary — it knows how to read refreshTime from the status block and compute secret_age_hours, but it has no opinion about whether 72 hours is acceptable. Rotation policy enforcement lives in run_pipeline via the ROTATION_POLICIES dict, which maps key-name prefixes to maximum-age thresholds. A key can be perfectly synced (ESO successfully wrote the latest value) yet still violate the rotation policy because the underlying credential in GCP Secret Manager is itself too old. Sync health and rotation compliance are orthogonal checks; the pipeline evaluates both (see Code Walkthrough).

The AlertDispatcher then receives a SecretSyncRecord and a derived severity — computed purely from the namespace, "critical" for prod and "warning" for staging — and routes to the appropriate channel. Keeping routing logic out of the evaluator means you can swap PagerDuty for OpsGenie, or add a new severity tier, without touching collection or evaluation code.

Environment-aware severity and rotation windows as policy

Treating the namespace as the severity signal — rather than hardcoding it into individual secret configurations — makes the policy consistent and auditable. Any secret in the prod namespace that fails either the sync check or its rotation window generates a critical alert. The same violation in staging generates a warning. This mirrors how most incident-response teams actually operate: production carries on-call weight; staging does not.

The per-provider rotation windows in ROTATION_POLICIES encode a real security posture: GCP service account keys ("gcp-sa-") rotate every 30 days because they are long-lived credentials with broad IAM scope; Anthropic API keys ("anthropic-") rotate every 60 days; OpenAI keys ("openai-") every 90 days. The pipeline enforces these windows continuously — not just at rotation time — so a key that was rotated on schedule but then left untouched drifts back into violation automatically when its age crosses the threshold (see Code Walkthrough for the prefix-matching loop in run_pipeline).

Code Walkthrough

Now that you understand the collector-evaluator-dispatcher pattern, here is how each component is implemented and wired together into a running pipeline.

The SyncStatusCollector queries the Kubernetes API for all ExternalSecret resources, reads the SecretSynced status condition that External Secrets Operator writes on each object, and produces a list of SecretSyncRecord dataclass instances carrying sync health, last-sync timestamp, and computed age in hours. The run_pipeline function then applies per-provider rotation policies inline via the ROTATION_POLICIES dict — 90 days for OpenAI keys, 60 days for Anthropic keys, 30 days for GCP service account keys — and the AlertDispatcher routes violations by environment and severity: production criticals to PagerDuty, production warnings to Slack #prod-secrets, and staging violations to #staging-alerts.

The following code defines the SecretSyncRecord dataclass and the SyncStatusCollector that populates it:

Code snippetpython
1from dataclasses import dataclass
2from datetime import datetime, timezone
3from typing import Optional
4from kubernetes import client, config
5
6@dataclass
7class SecretSyncRecord:
8    name: str
9    namespace: str
10    is_synced: bool
11    last_sync_time: Optional[datetime]
12    error_message: Optional[str]
13    secret_age_hours: float
14
15class SyncStatusCollector:
16    def __init__(self, kubeconfig_path: Optional[str] = None):
17        if kubeconfig_path:
18            config.load_kube_config(config_file=kubeconfig_path)
19        else:
20            config.load_incluster_config()
21        self.custom_api = client.CustomObjectsApi()
22
23    def collect_sync_status(self, namespaces: list[str]) -> list[SecretSyncRecord]:
24        records: list[SecretSyncRecord] = []
25        for ns in namespaces:
26            items = self.custom_api.list_namespaced_custom_object(
27                group="external-secrets.io",
28                version="v1beta1",
29                namespace=ns,
30                plural="externalsecrets",
31            ).get("items", [])
32            for item in items:
33                status = item.get("status", {})
34                conditions = status.get("conditions", [])
35                synced_cond = next(
36                    (c for c in conditions if c.get("type") == "SecretSynced"), None
37                )
38                is_synced = synced_cond.get("status") == "True" if synced_cond else False
39                error_msg = synced_cond.get("message") if not is_synced else None
40                last_sync_raw = status.get("refreshTime")
41                last_sync = (
42                    datetime.fromisoformat(last_sync_raw.replace("Z", "+00:00"))
43                    if last_sync_raw else None
44                )
45                age_hours = (
46                    (datetime.now(timezone.utc) - last_sync).total_seconds() / 3600
47                    if last_sync else float("inf")
48                )
49                records.append(SecretSyncRecord(
50                    name=item["metadata"]["name"],
51                    namespace=ns,
52                    is_synced=is_synced,
53                    last_sync_time=last_sync,
54                    error_message=error_msg,
55                    secret_age_hours=age_hours,
56                ))
57        return records

With the collector in place, the pipeline function applies rotation policies and dispatches alerts:

Code snippetpython
1ROTATION_POLICIES = {
2    "openai-":    90,   # max days before rotation required
3    "anthropic-": 60,
4    "gcp-sa-":    30,
5}
6
7def run_pipeline(collector, dispatcher) -> None:
8    records = collector.collect_sync_status(namespaces=["prod", "staging"])
9    for record in records:
10        severity = "critical" if record.namespace == "prod" else "warning"
11        if not record.is_synced:
12            dispatcher.send(record, severity=severity)
13            continue
14        for prefix, max_days in ROTATION_POLICIES.items():
15            if record.name.startswith(prefix) and record.secret_age_hours > max_days * 24:
16                dispatcher.send(record, severity=severity)
17                break

You'll know it works when production unsynced or overdue secrets appear as PagerDuty incidents and staging violations surface in #staging-alerts within one polling cycle, with no alert fired for secrets whose age falls within the policy window.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do read the SecretSynced status condition on each ExternalSecret object — that is the signal External Secrets Operator writes to surface sync failures from misconfigured SecretStore bindings or revoked IAM; checking only whether the downstream Kubernetes Secret resource exists will not detect a stalled sync and leaves workloads running on stale credentials.
2. ✓Do set secret_age_hours to float("inf") when last_sync_time is None — a record whose refreshTime field is absent has never completed a successful sync and must be treated as maximally overdue; any finite default would exempt it from every threshold check in run_pipeline and silently drop it from the alert pipeline.
3. ✓Do derive severity from record.namespace before calling dispatcher.send() — production secrets require PagerDuty incidents while staging violations route to #staging-alerts; collapsing all alerts into one channel buries critical production credential drift in low-urgency staging noise and defeats the environment-aware routing the dispatcher is built around.

Don'ts

1. ✗Don't rely on OpenAI or Anthropic inference failures to discover stale credentials — by the time a 401 surfaces at the inference layer, the SecretSynced condition on the ExternalSecret has already been False for an unknown window; SyncStatusCollector exists precisely to close that blind spot by detecting drift within one polling cycle.
2. ✗Don't skip the rotation policy loop when record.is_synced is True — run_pipeline checks both sync state and key age because a secret that synced successfully days ago can still violate the GCP SA 30-day window; treating a healthy SecretSynced=True as unconditionally compliant bypasses the entire ROTATION_POLICIES enforcement.
3. ✗Don't apply a single rotation threshold to all provider key prefixes — openai- keys allow 90 days, anthropic- 60 days, and gcp-sa- only 30 days; using any uniform window silently under-enforces the tightest policy and can leave GCP service account keys in circulation for up to three times longer than the 30-day limit before an alert fires.