Deploy change management with ArgoCD hooks

Introduction

In production GitOps workflows, nothing stops an unapproved commit from syncing to your cluster if there is no enforcement at the ArgoCD layer. When you leave approval checks as a social convention rather than a technical gate, freeze windows get skipped under pressure, emergency overrides go unlogged, and a failed sync leaves a change request dangling in "deploying" status with no record of what happened. By the end of this lesson, you'll be able to implement ArgoCD PreSync hooks that reject unapproved syncs and enforce freeze windows, handle emergency override annotations with an audit trail, and wire SyncFail hooks that automatically roll back a failed deployment and close the change management loop.

Key Terminology

• PreSync Hook — An ArgoCD lifecycle hook that executes a Kubernetes Job before any manifests are applied to the cluster; in this lesson, PresyncChangeValidator runs as a PreSync hook to gate every sync against the change management API, aborting the sync by exiting non-zero if validation fails.
• SyncFail Hook — An ArgoCD lifecycle hook that fires when a sync operation cannot bring resources to a healthy state; handle_sync_fail uses this hook to transition the change request to "failed" status and trigger automatic rollback using the stored pre_change_snapshot_id.
• Emergency Override Annotation — A metadata flag (emergency_override: "true") placed on the ArgoCD Application resource and read via ARGOCD_APP_ANNOTATIONS_emergency_override, allowing a sync to bypass change request validation while still writing an audit event so the override is never silent.
• Freeze Window — A scheduled time period during which all syncs are blocked at the hook layer regardless of prior approval; enforced by _check_freeze_windows, which re-validates freeze status at sync time so a change approved before a freeze was declared cannot slip through during the freeze.
• Change Request Lifecycle — The state machine a change request moves through — approved → deploying → deployed or failed — with the PreSync hook marking the deploying transition and either the PostSync or SyncFail hook closing it, ensuring no request is left dangling.
• Pre-change Snapshot — A point-in-time record of cluster state captured before a deployment and stored as pre_change_snapshot_id on the change request at creation time; the SyncFail hook passes this ID to the rollback API to restore the last known good configuration automatically.

Concepts

Hooks as Technical Gates, Not Social Conventions

ArgoCD exposes three hook points in its sync lifecycle — PreSync, Sync, and PostSync/SyncFail — that let you embed enforcement directly into the deployment path. A PreSync hook runs a Kubernetes Job before a single manifest is applied; if that Job exits non-zero, ArgoCD aborts the sync without touching any cluster resource. This exit-code contract is the entire mechanism behind PresyncChangeValidator: a missing or unapproved change request causes the script to exit 1, and ArgoCD treats that as a hard block — no partial apply, no side effects.

The architectural consequence is that compliance becomes structural rather than disciplinary. A freeze window skipped under pressure or an emergency override that goes unlogged are no longer possible through inattention alone; the hook must actively permit the sync, and permission requires either an approved change request or an override annotation that is itself written to the audit trail.

Sync-Time vs. Approval-Time Enforcement

An approval workflow that validates only at submission contains a timing gap: a change request approved at noon can sync at midnight during a freeze window declared in the interim, because the approval check ran hours before freeze conditions changed. Moving freeze window enforcement into the PreSync hook closes this gap — _check_freeze_windows queries the freeze API at the instant of each sync attempt, so the question "is right now a valid moment to deploy?" is answered separately from "should this change eventually deploy?", which was answered at approval time.

Keeping these two concerns in separate layers lets each evolve independently. You can tighten freeze schedules, add environment-specific windows, or temporarily declare an emergency freeze without touching the approval workflow at all. The PreSync hook simply re-asks the question on every sync.

Closing the Loop: Every "Deploying" State Must Exit

A change management system that leaves requests in "deploying" indefinitely becomes an unreliable source of truth. After an incident, you cannot tell which changes completed, which rolled back, and which simply stalled. The SyncFail hook closes this loop on the failure path: when ArgoCD cannot bring resources to a healthy state, the hook finds the change request still in "deploying", transitions it to "failed", and — if a pre_change_snapshot_id was stored at creation time — posts a rollback request to restore the last known good state automatically (see Code Walkthrough).

Emergency overrides fit the same closed-loop model. The PreSync hook logs the override as an audit event before permitting the sync, so every deployment — whether it went through normal approval or an emergency bypass — appears in the trail. A PostSync hook removes the annotation after a successful sync, preventing it from silently persisting into the next deployment. The result is an invariant: every change request that enters "deploying" status exits as either "deployed" or "failed", and every sync — approved or overridden — carries a provenance record.

Code Walkthrough

Now that you understand how emergency overrides, freeze-window enforcement, and SyncFail-driven rollback fit together conceptually, the following two hooks implement that closed loop at the GitOps layer.

The PreSync hook checks for the emergency override annotation first — if it is present, the event is logged to the audit trail and the sync is permitted without a matching change request. Otherwise it checks for active freeze windows, looks up the approved change request by commit SHA, and marks it "deploying" before exiting 0. Any check failure exits 1, aborting the sync.

Code snippetpython
1import os
2import sys
3import requests
4from datetime import datetime, timezone
5
6class PresyncChangeValidator:
7    def __init__(self, cm_api_url: str, api_token: str):
8        self.cm_api_url = cm_api_url.rstrip("/")
9        self.session = requests.Session()
10        self.session.headers["Authorization"] = f"Bearer {api_token}"
11
12    def validate(self, commit_sha: str, emergency_override: bool) -> None:
13        if emergency_override:
14            self._log_emergency_override(commit_sha)
15            return  # PostSync hook removes the annotation
16
17        self._check_freeze_windows()
18        cr_id = self._find_approved_change_request(commit_sha)
19        self._mark_deploying(cr_id)
20
21    def _check_freeze_windows(self) -> None:
22        resp = self.session.get(f"{self.cm_api_url}/freeze-windows/active")
23        resp.raise_for_status()
24        windows = resp.json().get("windows", [])
25        if windows:
26            raise ValueError(f"Blocked by freeze window: {windows[0]['name']}")
27
28    def _find_approved_change_request(self, commit_sha: str) -> str:
29        resp = self.session.get(
30            f"{self.cm_api_url}/change-requests",
31            params={"commit_sha": commit_sha, "status": "approved"},
32        )
33        resp.raise_for_status()
34        items = resp.json().get("items", [])
35        if not items:
36            raise ValueError(f"No approved change request for commit {commit_sha[:8]}")
37        return items[0]["id"]
38
39    def _mark_deploying(self, cr_id: str) -> None:
40        self.session.patch(
41            f"{self.cm_api_url}/change-requests/{cr_id}",
42            json={"status": "deploying", "started_at": datetime.now(timezone.utc).isoformat()},
43        ).raise_for_status()
44
45    def _log_emergency_override(self, commit_sha: str) -> None:
46        self.session.post(
47            f"{self.cm_api_url}/audit-events",
48            json={"event": "emergency_override", "commit_sha": commit_sha,
49                  "timestamp": datetime.now(timezone.utc).isoformat()},
50        )
51
52if __name__ == "__main__":
53    override = os.environ.get("ARGOCD_APP_ANNOTATIONS_emergency_override") == "true"
54    validator = PresyncChangeValidator(
55        cm_api_url=os.environ["CM_API_URL"],
56        api_token=os.environ["CM_API_TOKEN"],
57    )
58    try:
59        validator.validate(os.environ["ARGOCD_APP_REVISION"], emergency_override=override)
60        print("PreSync validation passed.")
61        sys.exit(0)
62    except ValueError as exc:
63        print(f"PreSync blocked: {exc}", file=sys.stderr)
64        sys.exit(1)

The SyncFail hook handles the other side of the loop. When ArgoCD cannot bring resources to a healthy state, this hook finds the change request that is still in "deploying" status, transitions it to "failed", and calls the rollback API with the pre-change snapshot ID that was stored on the change request at creation time — initiating an automatic revert to the last known good state.

Code snippetpython
1import os
2import sys
3import requests
4from datetime import datetime, timezone
5
6def handle_sync_fail(cm_api_url: str, api_token: str, commit_sha: str) -> None:
7    session = requests.Session()
8    session.headers["Authorization"] = f"Bearer {api_token}"
9
10    resp = session.get(
11        f"{cm_api_url}/change-requests",
12        params={"commit_sha": commit_sha, "status": "deploying"},
13    )
14    resp.raise_for_status()
15    items = resp.json().get("items", [])
16    if not items:
17        print("No deploying change request found; nothing to roll back.", file=sys.stderr)
18        return
19
20    cr = items[0]
21    session.patch(
22        f"{cm_api_url}/change-requests/{cr['id']}",
23        json={"status": "failed", "failed_at": datetime.now(timezone.utc).isoformat()},
24    ).raise_for_status()
25
26    snapshot_id = cr.get("pre_change_snapshot_id")
27    if snapshot_id:
28        session.post(
29            f"{cm_api_url}/rollbacks",
30            json={"change_request_id": cr["id"], "snapshot_id": snapshot_id},
31        ).raise_for_status()
32        print(f"Rollback initiated from snapshot {snapshot_id}.")
33    else:
34        print("No snapshot ID present; manual rollback required.", file=sys.stderr)
35
36if __name__ == "__main__":
37    handle_sync_fail(
38        cm_api_url=os.environ["CM_API_URL"],
39        api_token=os.environ["CM_API_TOKEN"],
40        commit_sha=os.environ["ARGOCD_APP_REVISION"],
41    )

Together these two hooks enforce the invariant from the Concepts section: every change request that reaches "deploying" status ends in either "deployed" (PostSync hook) or "failed" (SyncFail hook with automatic rollback) — never left dangling.

Confirm that a sync attempt with no matching approved change request causes ArgoCD to report a failed sync without applying any manifests, and that a deliberately broken manifest triggers the SyncFail hook and transitions the change request to "failed" status in your change management API.

Do's and Don'ts

Building on the hook implementations above, the following imperatives capture the failure modes most likely to bite when wiring these scripts into a real ArgoCD Application.

Do's

1. ✓Do exit 1 from the __main__ block whenever PresyncChangeValidator.validate() raises a ValueError — ArgoCD interprets any non-zero hook exit as a sync abort, so a hook that catches the exception but exits 0 is indistinguishable from a passing check and will let the unapproved commit apply manifests to the cluster.
2. ✓Do call _log_emergency_override before returning from validate() when the ARGOCD_APP_ANNOTATIONS_emergency_override annotation is present — the override deliberately skips both the freeze-window check and _find_approved_change_request, making the POST to /audit-events the only record that normal approval was bypassed; omitting it leaves emergency syncs completely invisible to the change management system.
3. ✓Do store pre_change_snapshot_id on each change request at creation time so handle_sync_fail can call the /rollbacks endpoint automatically — without it the SyncFail hook can only PATCH the request to "failed" and print a warning, leaving the cluster in the broken state and requiring a manual rollback that may arrive too late.

Don'ts

1. ✗Don't check freeze windows before evaluating the emergency_override flag — PresyncChangeValidator.validate() tests emergency_override first precisely because an active freeze window would block the sync before the override is ever read, defeating its purpose; reversing the order silently disables the emergency path during the periods it is most needed.
2. ✗Don't relax the commit_sha + status: approved filter in _find_approved_change_request — querying on SHA and status together ensures the exact revision ArgoCD is about to apply has been reviewed; dropping the SHA filter lets any currently approved change request satisfy the gate, allowing an unapproved commit to piggyback on a previous approval and reach the cluster unchecked.
3. ✗Don't allow a change request to remain in "deploying" status after a failed sync — handle_sync_fail must PATCH the status to "failed" and POST to /rollbacks in the same run; a dangling "deploying" row breaks the closed-loop invariant that every sync ends in either "deployed" or "failed", hiding the outage from on-call responders and the change management dashboard.