Deploy with blue-green Helm charts and atomic service switching

Introduction

When you promote an AI service from prototype to production, even a brief window of downtime during a release can affect every user hitting that endpoint. Traditional in-place updates replace the running version mid-request, making rollback slow and risky. Blue-green deployment sidesteps this by keeping two identical environments—one active, one idle—and switching traffic between them only after the new environment passes health gates. By the end of this lesson, you'll be able to implement a Helm-based blue-green pipeline that deploys to the idle slot, validates health, atomically switches the Kubernetes service selector, and preserves instant rollback capability.

Key Terminology

• Blue-green deployment — A release strategy that maintains two identical environments (called slots) in parallel, routing all production traffic to exactly one at a time, so the new version can be fully deployed and validated before any user sees it.
• Deployment slot — One of the two named color environments (blue or green) managed by BlueGreenDeployer; at any moment one slot is active (receiving live traffic) and the other is idle (available for the next release).
• Helm values — The configuration dictionary produced by generate_values that parameterizes a slot's container image, replica count, resource requests, and health check settings before a Helm chart is applied to that slot.
• Health gate — An async pre-cutover check (_check_health) that interrogates every pod in the target slot; if any pod fails, switch_traffic aborts and returns a failed SwitchResult without touching the active slot.
• Service selector patch — The Kubernetes API call inside _patch_service that atomically rewrites the slot label selector on the live service, redirecting all incoming requests from the old color to the new color in a single operation.
• Instant rollback — The property that the previously active slot remains fully deployed and reachable after a traffic switch, so reverting requires only another selector patch rather than a fresh deployment.

Concepts

The Two-Slot Model

Blue-green deployment replaces the conventional "update in place" approach with a parallel-environment strategy. Instead of mutating the running deployment—which exposes users to the transition window—you maintain two mirror environments, called slots, identified by color (blue and green). At any given moment exactly one slot holds live production traffic; the other is idle. A release cycle means deploying the new version into the idle slot, validating it thoroughly, and then flipping which slot is active. The old slot is never torn down immediately; it stays warm so that reverting is a configuration change, not a rebuild.

BlueGreenDeployer anchors this model in code. Its constructor calls _detect_active() to read the current Kubernetes service selector and discover which color is live. Every subsequent decision—which slot to deploy into, what to label as active in Helm values, which direction to "flip"—flows from that single source of truth stored in self.active_color.

Health-Gated Cutover

The safety guarantee of blue-green deployment rests on a strict sequencing rule: traffic never moves until the target slot has passed all health checks. In switch_traffic, the very first action is await self._check_health(target_color). If health.all_healthy is False—even a single pod failing—the method returns immediately with SwitchResult(success=False) and leaves self.active_color unchanged. The production slot keeps serving users and no rollback action is required because no switch occurred.

Only after every health check passes does switch_traffic proceed to the selector patch. This sequencing is what separates a health gate from a mere health check: the gate is a hard precondition that blocks the cutover, not a post-hoc observation. The pattern is analogous to a circuit breaker—fail fast, preserve the known-good state.

Atomic Traffic Switch via Service Selector

Kubernetes services route traffic to pods by matching labels. The key insight exploited here is that rewriting the service's spec.selector is a single API call that takes effect immediately across the entire cluster's routing layer—there is no rolling window, no half-old-half-new state. When _patch_service applies {"spec": {"selector": {"slot": target_color}}}, every subsequent connection is routed to the new slot; in-flight requests on existing connections complete against whichever pod they were already talking to (see Code Walkthrough).

This atomicity is what makes the idle slot a genuine instant-rollback target. Because the old slot's pods were never scaled down, re-applying the selector patch with the previous color restores full production behavior in the time it takes a single Kubernetes API call to propagate—seconds, not minutes.

Helm Values as Slot Configuration

Each slot gets its own Helm release parameterized by generate_values. The method sets service.active to True only when color == self.active_color, which controls whether the slot's pods are included in the service's endpoint set. All other fields—image tag, replica count, CPU/memory requests, health check path and timing—are slot-specific but structurally identical between the two colors, which is what makes the environments genuinely mirror slots rather than just two arbitrary deployments. Keeping the values symmetric is a prerequisite for traffic switching to be safe: if the idle slot has fewer replicas or different resource limits, the apparent health parity after switching may not reflect actual capacity.

Code Walkthrough

Now that you understand the blue-green model—two parallel slots where exactly one receives live traffic at a time—the following Python implementation shows how to automate slot deployment and traffic switching using Helm.

The BlueGreenDeployer class manages the full lifecycle. Its constructor discovers which color slot is currently active by querying the Kubernetes service selector, storing that state for downstream decisions. The generate_values method assembles the Helm values dictionary for a specific slot: it embeds the container image repository and tag, sets a default replica count of three, declares CPU and memory resource requests, marks the service active flag only when this slot matches the live color, and configures the health check endpoint and probe timing:

Code snippetpython
1class BlueGreenDeployer:
2    """Manages blue-green deployments via Helm."""
3
4    def __init__(self, chart_path: str, namespace: str):
5        self.chart_path = chart_path
6        self.namespace = namespace
7        self.active_color = self._detect_active()
8
9    def generate_values(
10        self, color: str, image_tag: str, config: dict
11    ) -> dict:
12        """Build Helm values for a deployment slot."""
13        return {
14            "slot": color,
15            "image": {
16                "repository": config["repository"],
17                "tag": image_tag,
18            },
19            "replicas": config.get("replicas", 3),
20            "resources": {
21                "requests": {
22                    "cpu": config.get("cpu", "500m"),
23                    "memory": config.get("mem", "1Gi"),
24                },
25            },
26            "service": {
27                "active": color == self.active_color,
28            },
29            "healthCheck": {
30                "path": "/health",
31                "initialDelay": 30,
32                "period": 10,
33            },
34        }

Once the idle slot is deployed and all pods are running, switch_traffic performs the cutover. It first runs async health checks against every pod in the target slot; if any check fails it aborts immediately and returns a failure result—leaving the active slot untouched. Only when all health gates pass does it apply a Kubernetes service patch that rewrites the pod selector to the new color, making the switch atomic from the perspective of incoming requests:

Code snippetpython
1from dataclasses import dataclass, field
2from typing import List
3
4@dataclass
5class SwitchResult:
6    success: bool
7    reason: str = ""
8    previous: str = ""
9    current: str = ""
10    details: List[str] = field(default_factory=list)
11
12async def switch_traffic(self, target_color: str) -> SwitchResult:
13    """Switch production traffic to target color."""
14    health = await self._check_health(target_color)
15    if not health.all_healthy:
16        return SwitchResult(
17            success=False,
18            reason="Health check failed",
19            details=health.failures,
20        )
21    patch = {"spec": {"selector": {"slot": target_color}}}
22    self._patch_service(f"ai-service-{self.namespace}", patch)
23    self.active_color = target_color
24    return SwitchResult(
25        success=True,
26        previous=self._opposite(target_color),
27        current=target_color,
28    )

The old environment continues serving traffic right up until _patch_service rewrites the selector. If the new environment fails health checks at any point before that call, the switch never happens and no separate rollback action is needed—the idle slot simply stays idle while production traffic flows uninterrupted through the original slot.

Confirm that after calling switch_traffic with a fully healthy target slot, SwitchResult.success is True and SwitchResult.current matches the target color, while the previously active slot remains reachable at its internal endpoint for instant rollback if needed.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do gate every _patch_service call on health.all_healthy being True — switch_traffic runs async health checks against every pod in the target slot and returns a failing SwitchResult without touching the selector if any pod fails; this is the only mechanism that keeps the original slot serving uninterrupted traffic while the idle slot is broken.
2. ✓Do set service.active in generate_values using the expression color == self.active_color — this boolean drives which Kubernetes pod selector is marked live; if the wrong slot gets active: true at deploy time, the Helm release can overwrite the selector before switch_traffic is even called, bypassing the health gate entirely.
3. ✓Do confirm SwitchResult.success is True and SwitchResult.current matches the target color before treating a switch as complete — _patch_service mutates self.active_color in memory only on success, so an unchecked failed switch leaves the in-process state inconsistent with the actual Kubernetes selector, causing the next deployment to target the wrong idle slot.

Don'ts

1. ✗Don't destroy or redeploy the previously active slot the moment switch_traffic returns success=True — the old color remains reachable at its internal endpoint specifically because rollback is instant only while that slot is still running; tearing it down converts a one-line _patch_service rollback into a full Helm re-deploy under live pressure.
2. ✗Don't hard-code which color is active when constructing BlueGreenDeployer — the constructor calls _detect_active() to read the real Kubernetes service selector, because a prior switch run outside this deployer's lifecycle can leave the cluster in either color; assuming a fixed starting color means generate_values sets service.active on the wrong slot and switch_traffic patches the selector in the wrong direction.
3. ✗Don't skip the initialDelay: 30 and period: 10 probe timing in generate_values when adjusting the health check path — the /health gate in switch_traffic depends on probes having had enough time to cycle; shortening or omitting initialDelay causes pods to report healthy before the AI service process has fully loaded its model weights, letting a not-yet-ready slot pass all_healthy and go live.