Deploy Argo Rollouts with Canary Strategy for LiteLLM Model Config Changes

Introduction

In production, switching a LiteLLM routing configuration from one model to another by swapping a standard Kubernetes Deployment carries significant risk — a misconfigured model or degraded response quality goes live for every user simultaneously, with no safe rollback path short of a full redeployment. Argo Rollouts solves this by replacing the Deployment primitive with a Rollout CRD that supports multi-step canary progressions, automated analysis gates, and instant rollback to any recent revision. By the end of this lesson, you'll be able to install the Argo Rollouts controller, define a canary Rollout for LiteLLM, and configure traffic-weight steps that protect production while a new model configuration proves itself under real load.

Key Terminology

• Rollout CRD — The rollouts.argoproj.io/v1alpha1 custom resource that replaces a standard Kubernetes Deployment, giving the Rollouts Controller authority to govern traffic promotion across canary weight steps rather than letting pod readiness alone drive the traffic shift.
• canary progression — The ordered sequence of setWeight, pause, and analysis entries under strategy.canary.steps that incrementally advances traffic to the canary ReplicaSet (10 → 25 → 50 → 100 percent) while enforcing timed holds and analysis gates at each threshold.
• stable / canary Service — Two Kubernetes Services (litellm-stable and litellm-canary) whose pod selectors the Rollouts Controller rewrites at each step to bind each service to its corresponding ReplicaSet; declared via the stableService and canaryService fields in the Rollout spec.
• promotion gate — A per-step analysis entry in the canary steps list that blocks advancement to the next setWeight until the referenced AnalysisRun (e.g., llm-quality-analysis) returns a passing verdict; a failing gate aborts the rollout and restores 100% traffic to the stable fleet.
• background analysis — The top-level analysis block with startingStep: 2 that runs a continuous AnalysisRun from a configured step onward, independent of step transitions, and can abort the rollout at any point if quality degrades under sustained load.
• rollbackWindow — A Rollout field (rollbackWindow.revisions: 3) that retains a configurable number of recent stable ReplicaSets on the cluster so traffic can be instantly redirected to a prior revision without triggering a full redeployment.

Concepts

Why the Deployment Primitive Fails for Model Config Changes

A standard Kubernetes Deployment performs rolling updates by replacing old pods with new ones as they pass their readiness probe — but traffic shift during that transition is an implementation detail of pod scheduling, not a controlled policy. The moment new pods become ready, the Deployment's load balancer starts routing real user traffic to them. For a LiteLLM routing configuration change — swapping which model the gateway routes to — this means a misconfigured model or silently degraded response quality reaches 100% of production traffic the moment the pod fleet rolls over. There is no pause for observation, no automated quality gate, and no rollback path short of a full redeployment that opens its own failure window.

Argo Rollouts solves this by replacing the Deployment with a Rollout CRD. The Rollouts Controller watches these resources and interposes itself between a spec change and pod promotion — holding traffic at each weight threshold until either a timed pause expires or an AnalysisRun confirms the canary meets quality criteria.

The Two-Service Traffic Split Architecture

Canary progression requires two distinct traffic paths: one always pointing at the stable pod fleet and one pointing at the canary fleet. Argo Rollouts models this as two Kubernetes Services — litellm-stable and litellm-canary — whose pod selectors it rewrites automatically at each step. You do not manage the selectors directly; the Rollouts Controller mutates them as it creates and scales the canary ReplicaSet. The NGINX ingress controller reads weight annotations that the controller writes to the stableIngress resource to divide inbound traffic at the configured ratio.

This is why both Services must exist before the Rollout is applied (see Code Walkthrough). Pre-creating empty Services lets the controller take ownership of their selectors on first apply without racing against initial pod creation.

Promotion Gates vs. Background Analysis

The canary steps list contains two analytically distinct entry types. Per-step analysis blocks are promotion gates: when the rollout reaches that step, it creates an AnalysisRun and waits for a passing verdict before advancing to the next setWeight. A failing gate immediately aborts the progression and returns traffic to the stable fleet.

The top-level analysis block with startingStep: 2 is a separate mechanism — a continuous background monitor that activates at the specified step and remains active throughout the rest of the rollout, independent of step transitions. It can terminate the rollout at any moment after step 2 without requiring a step boundary to trigger it. This two-layer design catches distinct failure modes: per-step gates validate quality at each traffic threshold, while background monitoring catches slow-burn degradation that only becomes statistically significant after sustained load at a given weight (see Code Walkthrough).

Code Walkthrough

Now that you understand why the Deployment primitive fails for model-config changes, how the two-Service traffic split routes stable and canary traffic, and how promotion gates and background analysis govern each weight step, the next step is to install Argo Rollouts and define the Rollout manifest that drives a canary progression for LiteLLM model configuration changes.

Install the Argo Rollouts controller via Helm, then confirm the controller pod and the required CRDs are registered before proceeding:

Code snippetbash
1helm repo add argo https://argoproj.github.io/argo-helm
2helm repo update
3
4helm install argo-rollouts argo/argo-rollouts \
5  --namespace argo-rollouts \
6  --create-namespace \
7  --set dashboard.enabled=true \
8  --set controller.resources.requests.cpu=200m \
9  --set controller.resources.requests.memory=256Mi
10
11kubectl get pods -n argo-rollouts
12kubectl get crd | grep argoproj

You should see the rollouts.argoproj.io and analysistemplates.argoproj.io CRDs registered, with the controller pod reaching Running state within about 30 seconds.

With the controller running, define a Rollout resource that shifts traffic across four weight steps — 10%, 25%, 50%, and 100% — pausing at each step for analysis before advancing. The two companion Services (litellm-stable and litellm-canary) must exist before the Rollout is applied; Argo Rollouts rewrites their selectors during progression to bind each service to the appropriate ReplicaSet:

Code snippetyaml
1apiVersion: argoproj.io/v1alpha1
2kind: Rollout
3metadata:
4  name: litellm-model-rollout
5  namespace: genai-platform
6spec:
7  replicas: 3
8  revisionHistoryLimit: 5
9  selector:
10    matchLabels:
11      app: litellm-gateway
12  strategy:
13    canary:
14      canaryService: litellm-canary
15      stableService: litellm-stable
16      trafficRouting:
17        nginx:
18          stableIngress: litellm-ingress
19      steps:
20        - setWeight: 10
21        - pause: {duration: 10m}
22        - analysis:
23            templates:
24              - templateName: llm-quality-analysis
25        - setWeight: 25
26        - pause: {duration: 15m}
27        - analysis:
28            templates:
29              - templateName: llm-quality-analysis
30        - setWeight: 50
31        - pause: {duration: 20m}
32        - analysis:
33            templates:
34              - templateName: llm-quality-analysis
35        - setWeight: 100
36      analysis:
37        templates:
38          - templateName: llm-quality-analysis
39        startingStep: 2
40      rollbackWindow:
41        revisions: 3
42  template:
43    metadata:
44      labels:
45        app: litellm-gateway
46    spec:
47      containers:
48        - name: litellm
49          image: ghcr.io/berriai/litellm:main-latest
50          env:
51            - name: MODEL_CONFIG_VERSION
52              value: "v2.1.0"
53          ports:
54            - containerPort: 4000
55---
56apiVersion: v1
57kind: Service
58metadata:
59  name: litellm-stable
60  namespace: genai-platform
61spec:
62  selector:
63    app: litellm-gateway
64  ports:
65    - port: 4000
66      targetPort: 4000
67---
68apiVersion: v1
69kind: Service
70metadata:
71  name: litellm-canary
72  namespace: genai-platform
73spec:
74  selector:
75    app: litellm-gateway
76  ports:
77    - port: 4000
78      targetPort: 4000

The canaryService and stableService fields tell the Rollouts Controller which Service to pin to the canary ReplicaSet and which to keep bound to stable pods. The top-level analysis block with startingStep: 2 activates the llm-quality-analysis AnalysisRun as continuous background monitoring beginning at the third step, while the per-step analysis entries act as explicit promotion gates at each weight threshold. Setting rollbackWindow.revisions to 3 retains the last three stable revisions so you can revert without a full redeployment.

Confirm that kubectl argo rollouts get rollout litellm-model-rollout -n genai-platform shows the rollout in a Healthy or Paused state and that both litellm-stable and litellm-canary services list selectors that the controller is actively managing.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do pre-create both litellm-stable and litellm-canary Services before applying the Rollout manifest — Argo Rollouts rewrites their pod selectors during progression to bind each to the correct ReplicaSet, and a missing Service causes the controller to stall at initialization rather than failing with a clear error.
2. ✓Do verify that rollouts.argoproj.io and analysistemplates.argoproj.io CRDs are registered via kubectl get crd | grep argoproj before deploying the Rollout resource — applying the Rollout manifest against a cluster where the Helm install hasn't completed registration returns a "no kind Rollout is registered" API error that looks like a manifest bug but is actually a controller timing issue.
3. ✓Do set rollbackWindow.revisions: 3 and use the two-tier analysis pattern — per-step analysis gates plus the top-level analysis block with startingStep: 2 — the per-step gates enforce explicit promotion checkpoints at 10%, 25%, and 50% weight while the background llm-quality-analysis AnalysisRun catches degradation between steps, so neither layer alone provides the same protection as both together.

Don'ts

1. ✗Don't point canaryService and stableService at the same Service object — the Rollouts Controller rewrites each Service's selector independently to separate canary pods from stable pods; sharing a single Service causes both traffic pools to collapse into one, making the setWeight steps meaningless and sending 100% of traffic to whichever ReplicaSet wins the selector race.
2. ✗Don't skip the pause durations between setWeight steps — removing the pause: {duration: 10m} / 15m / 20m intervals causes the controller to advance through weight thresholds faster than the llm-quality-analysis AnalysisRun can collect enough LiteLLM response samples to produce a statistically meaningful verdict, so the analysis gate passes vacuously even when the new model config is degraded.
3. ✗Don't use a standard Kubernetes Deployment manifest alongside the Rollout for the same app: litellm-gateway selector — Argo Rollouts takes ownership of the ReplicaSets matched by the Rollout's selector.matchLabels; a competing Deployment will fight the controller over replica counts and selector rewrites, producing unpredictable traffic splits and breaking the canary progression entirely.