Build Helm charts with Skaffold local development workflow

Introduction

When you change a single line of Python in a GenAI service, the path to seeing that change run inside Kubernetes is brittle: build the image, push it to a registry, bump the chart, run helm upgrade, wait for the pod, repeat. Teams that skip a deliberate inner loop here lose hours per day to manual ceremony, and when prod values silently drift from dev values, "works on my cluster" becomes a real outage. By the end of this lesson you'll be able to structure a Helm chart with environment-specific values for an AI service and wire Skaffold so a code edit reaches a running pod in under thirty seconds.

Key Terminology

• Helm chart — a packaged set of templated Kubernetes manifests plus a values schema; in this lesson it is the unit you parameterize per environment so dev, staging, and prod share one source of truth.
• Values file — the YAML inputs (values.yaml, values-dev.yaml, …) Helm renders the templates against; this is where dev's cheap model and prod's autoscaling rules diverge cleanly.
• Skaffold — a CLI that watches your source tree and re-runs build → push → helm upgrade automatically; it is what collapses the manual cycle into one command (skaffold dev).
• Inner dev loop — the elapsed time from "save file" to "request hits new pod"; the metric this lesson optimizes (target: under thirty seconds for Python services).
• File sync — Skaffold's optimization that copies changed files into a running container instead of rebuilding the image, used for interpreted code paths.

Concepts

Helm chart layout for a GenAI service

A Helm chart is a directory with a fixed shape: Chart.yaml declares the chart, values.yaml holds defaults, per-environment files override only what differs, and templates/ holds the Kubernetes resources rendered against those values.

For an AI service this layout matters because the things that vary across environments are exactly the levers you want to tune: model name (Gemini Flash in dev, GPT-4o in prod), replica count, resource limits, and whether the HPA renders at all. One template set, three values files, no copy-paste drift.

Environment-specific values

Each values file overrides only the fields it needs. values-dev.yaml might pin replicaCount: 1 and a cheap model; values-prod.yaml enables HPA and raises memory limits. Helm merges them — defaults from values.yaml first, then the environment file wins on conflicts. This is what makes "promote this change from dev to prod" a one-line Skaffold profile switch instead of a hand-edited YAML diff.

Skaffold's inner loop

Skaffold sits on top of Helm and turns the manual build/push/upgrade chain into a file watcher. When you save a Python file, Skaffold rebuilds the image (or syncs the file directly into the pod for interpreted code), pushes only the changed layer, and runs helm upgrade against your dev values. The loop closes in seconds — see Code Walkthrough for the wiring.

Code Walkthrough

The snippet below wires the previous concepts together: it generates a Chart.yaml plus a values-dev.yaml for an AI service, and emits the skaffold.yaml that points Skaffold at that chart for hot-reload dev with Python file sync.

Code snippetpython
1from pydantic import BaseModel, Field, field_validator
2import yaml
3
4class LLMConfig(BaseModel):
5    provider: str = "openai"
6    model: str = "gpt-4o"
7    max_tokens: int = 4096
8
9class ChartMetadata(BaseModel):
10    name: str = "ai-service"
11    version: str = "0.1.0"
12    description: str = "Helm chart for GenAI service"
13    api_version: str = Field(default="v2", alias="apiVersion")
14
15    @field_validator("name")
16    @classmethod
17    def lowercase_hyphens(cls, v: str) -> str:
18        if "_" in v or v != v.lower():
19            raise ValueError("Chart name must be lowercase with hyphens")
20        return v
21
22class DevValues(BaseModel):
23    replicaCount: int = 1
24    image: dict = {"repository": "gcr.io/project/ai-service", "tag": "dev"}
25    llm: LLMConfig = LLMConfig(provider="gemini", model="gemini-2.0-flash")
26    hpa: dict = {"enabled": False}
27
28def write_yaml(path: str, model: BaseModel) -> None:
29    with open(path, "w") as f:
30        yaml.dump(model.model_dump(by_alias=True), f, default_flow_style=False)
31
32def write_skaffold(path: str = "skaffold.yaml") -> None:
33    cfg = {
34        "apiVersion": "skaffold/v4beta6",
35        "kind": "Config",
36        "build": {
37            "artifacts": [{
38                "image": "gcr.io/project/ai-service",
39                "context": ".",
40                "docker": {"dockerfile": "Dockerfile"},
41                "sync": {"infer": ["**/*.py"]},
42            }]
43        },
44        "deploy": {
45            "helm": {
46                "releases": [{
47                    "name": "ai-service",
48                    "chartPath": "./charts/ai-service",
49                    "valuesFiles": ["./charts/ai-service/values-dev.yaml"],
50                    "namespace": "ai-dev",
51                }]
52            }
53        },
54    }
55    with open(path, "w") as f:
56        yaml.dump(cfg, f, default_flow_style=False)
57
58if __name__ == "__main__":
59    write_yaml("charts/ai-service/Chart.yaml", ChartMetadata())
60    write_yaml("charts/ai-service/values-dev.yaml", DevValues())
61    write_skaffold()

ChartMetadata enforces Helm's naming rule (chart names appear in Kubernetes labels, so underscores and uppercase are illegal) and emits apiVersion rather than the snake-cased Python attribute. DevValues overrides only what dev needs — Gemini Flash for cheap iteration and HPA off because one replica is plenty. write_skaffold wires Skaffold to that chart and uses sync.infer so .py edits skip the image rebuild and copy straight into the running pod.

You'll know it works when skaffold dev boots the chart in the ai-dev namespace and a save to any *.py file shows up as a new request response in under thirty seconds without a docker build line appearing in the Skaffold log.

Do's and Don'ts

Do's

1. ✓Do keep one chart and per-env values files — the chart is the contract; environments are inputs. Drift dies here.
2. ✓Do enable Skaffold file sync for Python paths — sync.infer: ["**/*.py"] collapses the loop to seconds for code-only edits.
3. ✓Do gate prod-only resources behind values flags — hpa.enabled: false in dev keeps the rendered manifest clean and avoids confusing autoscaling behavior locally.

Don'ts

1. ✗Don't copy-paste manifests per environment — every divergence becomes a future incident; let Helm merge defaults with overrides.
2. ✗Don't run skaffold dev against prod values — it will happily helm upgrade your prod release on every save. Pin a dev profile and the dev namespace.
3. ✗Don't bake secrets into values-*.yaml — values files are checked in; secrets belong in a separate, untracked source (sealed-secrets, External Secrets, etc.).