Implement blue-green and progressive delivery with Argo Rollouts

Introduction

In production, rolling updates offer no validation gate between "the new version is starting" and "it is handling 100% of traffic" — for a GenAI inference service, a single tokenization regression can corrupt every response in flight before anyone notices. Blue-green deployments keep the old ReplicaSet fully running while the new one warms up, then switch traffic atomically only after automated checks pass. This lesson teaches you how to configure an Argo Rollouts Rollout CRD with a blue-green strategy, wire up an AnalysisTemplate as a pre-promotion SLO gate, and use scaleDownDelaySeconds to preserve instant rollback capability.

Key Terminology

• Rollout: An Argo Rollouts CRD that supersedes Deployment, supporting blue-green and canary strategies natively.
• AnalysisTemplate / AnalysisRun: A reusable definition of "did this deploy succeed?" via Prometheus, Datadog, or shell-script probes.
• Active Service / Preview Service: Two K8s Services. Active points at the live ReplicaSet (blue); Preview points at the new one (green) for pre-cut-over validation.
• autoPromotionEnabled: When true, the new version is promoted automatically after analysis passes; when false, requires manual kubectl argo rollouts promote.
• scaleDownDelaySeconds: How long the old ReplicaSet stays running after promotion, providing instant rollback capability.

Concepts

Operating discipline

1. Always require manual promotion in production. The 30 seconds it takes to type argo rollouts promote is a forcing function for the engineer to actually look at the dashboards.
2. Set scaleDownDelaySeconds to at least 10 minutes. Anything less than that and rollback becomes a re-deploy from the previous tag — slower and riskier.
3. Use the preview Service for synthetic checks. Run the team's evaluation suite against the preview URL before promoting; that's the per-deploy gate that catches prompt regressions.
4. ✗Don't blue-green stateful workloads. Blue-green's clean cut-over assumes stateless replicas. Anything with local state needs the expand-and-contract migration pattern instead.
5. Pin the AnalysisTemplate's success thresholds in Git, not in dashboards. The threshold is part of the deploy contract and should live alongside the Rollout manifest.

Code Walkthrough

Now that you understand the blue-green operating discipline — manual promotion, a 10-minute scale-down delay, the preview Service as the analysis target, and SLO thresholds pinned in Git — here is how those principles translate into manifests.

The Rollout below replaces a standard Deployment for the qa-bot service. Two Services (qa-bot-active and qa-bot-preview) must already exist in the cluster; the Argo Rollouts controller manipulates their label selectors atomically at promotion time. autoPromotionEnabled: false means the cut-over to green always requires a manual kubectl argo rollouts promote qa-bot, giving the on-call engineer time to review dashboards. scaleDownDelaySeconds: 600 keeps the old blue ReplicaSet running for 10 minutes after promotion so a single kubectl argo rollouts undo flips traffic back instantly if green misbehaves under real load. The prePromotionAnalysis block names the AnalysisTemplate that runs against green via the preview Service before any traffic is switched.

Code snippetyaml
1apiVersion: argoproj.io/v1alpha1
2kind: Rollout
3metadata:
4  name: qa-bot
5spec:
6  replicas: 6
7  strategy:
8    blueGreen:
9      activeService: qa-bot-active
10      previewService: qa-bot-preview
11      autoPromotionEnabled: false
12      scaleDownDelaySeconds: 600
13      prePromotionAnalysis:
14        templates:
15          - templateName: latency-p95
16        args:
17          - name: service
18            value: qa-bot
19  selector:
20    matchLabels: {app: qa-bot}
21  template:
22    metadata:
23      labels: {app: qa-bot}
24    spec:
25      containers:
26        - name: app
27          image: us-docker.pkg.dev/platform/services/qa-bot:2.7.0

Define the AnalysisTemplate in the same Git commit as the Rollout so the success threshold is part of the deploy contract, not a dashboard setting that can drift:

Code snippetyaml
1apiVersion: argoproj.io/v1alpha1
2kind: AnalysisTemplate
3metadata:
4  name: latency-p95
5spec:
6  args:
7    - name: service
8  metrics:
9    - name: p95-latency
10      successCondition: result[0] < 1.5
11      failureLimit: 2
12      interval: 30s
13      count: 6
14      provider:
15        prometheus:
16          address: http://prometheus.monitoring:9090
17          query: |
18            histogram_quantile(0.95,
19              sum(rate(http_request_duration_seconds_bucket{
20                service="{{args.service}}",
21                version="green"
22              }[2m]))
23              by (le)
24            )

Six 30-second samples produce a 3-minute analysis window. The version="green" label filter isolates the new ReplicaSet's metrics from the still-live blue pods so the P95 judgment is uncontaminated by blue's baseline traffic. failureLimit: 2 tolerates a single transient spike; two consecutive failures abort the promotion automatically and scale green back to zero, leaving blue serving all traffic unchanged.

Confirm that the rollout is wired correctly by running kubectl argo rollouts get rollout qa-bot --watch after pushing a new image tag — the status should progress through Progressing, then Paused while the pre-promotion analysis runs, then Paused again waiting for your manual promote command, and finally Healthy after you run kubectl argo rollouts promote qa-bot; if the AnalysisRun fails, the status shows Degraded and blue continues serving traffic with no intervention required.

Do's and Don'ts

Having walked through implementing blue-green and progressive delivery with Argo Rollouts above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do co-commit the AnalysisTemplate in the same Git commit as the Rollout — pinning successCondition: result[0] < 1.5 in version control makes the P95 latency threshold part of the deploy contract; a threshold that lives only in a dashboard can be silently changed between deployments, making the pre-promotion gate meaningless.
2. ✓Do include the version="green" label filter in your Prometheus query — without it, the histogram_quantile aggregation mixes metrics from the still-live blue pods with green's, diluting any latency regression in the new ReplicaSet and allowing a broken tokenization path to pass the prePromotionAnalysis check undetected.
3. ✓Do keep scaleDownDelaySeconds: 600 at or above your team's mean rollback decision time — the 10-minute window is what makes kubectl argo rollouts undo an instant label-selector flip rather than a cold pod start; if blue scales down before you confirm green is healthy under real load, rollback becomes a multi-minute ReplicaSet creation instead of a sub-second traffic switch.

Don'ts

1. ✗Don't set autoPromotionEnabled: true on a GenAI inference Rollout — auto-promotion cuts traffic over to green the moment the ReplicaSet is ready, before prePromotionAnalysis has run its full 3-minute window; a tokenization regression that only appears under load pressure will corrupt 100% of responses before the failureLimit: 2 threshold can abort the promotion.
2. ✗Don't define the prePromotionAnalysis AnalysisTemplate separately from the Rollout manifest — deploying the Rollout first with a missing or mismatched latency-p95 template causes the controller to skip or error the analysis phase entirely, leaving the promotion gate silently open and the P95 SLO unenforced at the most critical moment.
3. ✗Don't lower failureLimit to 0 expecting stricter safety — a single transient Prometheus scrape gap or cold-start spike will abort promotion of a healthy green build; the failureLimit: 2 / interval: 30s / count: 6 combination is calibrated so one spike is tolerated while two consecutive failures — the signature of a real regression — trigger automatic scale-down of green and restoration of blue as the sole traffic target.