Manage deployment lifecycle with kubectl rollout

Introduction

When you push a new image of an LLM chat service to production and the new pods start crash-looping on a misconfigured proxy URL, you need to know — within seconds — whether the rollout is stuck, how to pause it, and how to revert. The kubectl rollout command family is the lever for every one of those decisions. Teams that skip it often discover a broken rollout only after users notice timeouts, by which point both old and new ReplicaSets are degraded and recovery requires guesswork. By the end of this lesson you'll be able to drive a Deployment update through its full lifecycle — status, pause, resume, restart, undo — and intervene safely when an update goes sideways.

Key Terminology

• rollout — the controller-managed transition from one Deployment revision to the next; every kubectl rollout subcommand operates on this state machine.
• revision — an immutable snapshot of a Deployment's pod template; rollout history lists them and rollout undo targets them by number.
• progressDeadlineSeconds — the Deployment field that decides when a stuck rollout is declared failed; it's what makes rollout status exit non-zero in CI.
• change-cause annotation — kubernetes.io/change-cause set on a Deployment so each revision carries a human-readable description in rollout history.
• rolling restart — a rollout restart that bumps a restartedAt annotation in the pod template, forcing a controlled pod recycle without a spec change.

Concepts

The rollout state machine

A Deployment update isn't a single event — it's a state machine the controller drives by scaling a new ReplicaSet up and the old one down. kubectl rollout exposes five verbs against that state machine: status (observe), history (audit), pause/resume (gate), restart (recycle without a spec change), and undo (revert). Every production change to an LLM service goes through this loop, so the verbs aren't optional knowledge — they're the control surface (see Code Walkthrough).

Status, history, and the failure signal

rollout status blocks until the new ReplicaSet reaches its desired count or progressDeadlineSeconds elapses, then exits 0 or non-zero. That exit code is the only honest signal a CI pipeline gets — kubectl apply returns success the moment the API accepts the manifest, long before any pod is healthy. rollout history complements it by recording every revision; pair it with the kubernetes.io/change-cause annotation so each entry carries why the change happened, not just that it did. Revisions that only change replica count don't appear, because the pod template is unchanged.

Pause, resume, restart, undo

Pausing freezes the rollout mid-update — the new ReplicaSet stops scaling up, the old one stops scaling down, and the Service load-balances across both. That gives you a zero-infrastructure canary: pause after the first new pod, watch logs and latency, then resume or undo. restart is the odd one out: it doesn't change the spec, it stamps a restartedAt annotation so pods are recycled in a rolling fashion — exactly what you want after rotating a Secret-mounted API key, since Kubernetes won't restart pods automatically when a Secret changes. undo rolls back to the prior revision (or --to-revision=N) by re-promoting the old ReplicaSet, which is fast because the old pods may still be present at zero replicas.

Code Walkthrough

This snippet demonstrates the full state-machine loop from the Concepts section: apply, observe with status, audit with history, intervene with pause/resume/restart, and revert with undo — all against an LLM chat Deployment.

Code snippetbash
1#!/usr/bin/env bash
2set -euo pipefail
3
4DEPLOY=deployment/llm-chat
5
6kubectl apply -f llm-chat-deployment.yaml
7kubectl annotate "$DEPLOY" \
8  kubernetes.io/change-cause="v1.2.0: new Gemini timeout config" --overwrite
9
10if ! kubectl rollout status "$DEPLOY" --timeout=120s; then
11  echo "rollout stalled — pausing for inspection"
12  kubectl rollout pause "$DEPLOY"
13  kubectl rollout history "$DEPLOY"
14  kubectl logs -l app=llm-chat --tail=50
15
16  read -rp "resume (r) or undo (u)? " choice
17  case "$choice" in
18    r) kubectl rollout resume "$DEPLOY" ;;
19    u) kubectl rollout undo "$DEPLOY" ;;
20  esac
21  exit 1
22fi
23
24kubectl rollout restart "$DEPLOY"
25kubectl rollout status "$DEPLOY" --timeout=120s

Lines 6–8 push the new spec and stamp a change-cause so the next rollout history entry is self-documenting. Line 10 turns the apply into a CI-grade gate: the script only proceeds on a healthy rollout. Lines 11–22 are the human-in-the-loop branch — pause first (so the cluster doesn't keep churning while you investigate), then either resume or undo. The trailing restart (line 24) shows the second use case for the verb: bouncing pods to pick up a freshly rotated Secret without editing the manifest.

You'll know it works when kubectl rollout status deployment/llm-chat exits with code 0 on the green path, and when forcing a bad image makes the script pause and prompt instead of leaving the Deployment in a half-rolled state.

Do's and Don'ts

Do's

1. ✓Do gate CI on rollout status --timeout=… — kubectl apply succeeds before any pod is healthy, so without the timeout your pipeline reports green on a stuck rollout.
2. ✓Do annotate every change with kubernetes.io/change-cause — rollout history is only useful if each revision tells you why it exists.
3. ✓Do rollout restart after rotating a Secret — Kubernetes does not restart pods when a mounted Secret changes; the running pods keep using the old value until you bounce them.

Don'ts

1. ✗Don't leave a Deployment paused — a paused rollout blocks every future update silently; if you pause to inspect, decide and either resume or undo before walking away.
2. ✗Don't rollout undo without checking rollout history --revision=N first — the previous revision may itself be the broken one; verify the target spec before rolling back into it.
3. ✗Don't expect kubectl scale to appear in rollout history — replica-count changes don't create revisions, so don't rely on history to audit them.