Use Kustomize bases and overlays for the LLM app

Introduction

When you ship the same LLM chat stack to dev, staging, and prod, you end up copy-pasting near-identical YAML and editing replica counts, image tags, and namespaces by hand — and one bad sed run rolls the wrong model image into production. Kustomize fixes this by keeping ONE set of base manifests and layering small per-environment patches on top, with no template syntax. By the end of this lesson you'll be able to lay out a base/ + overlays/{dev,staging,prod}/ tree for the LLM chat stack, write a base kustomization.yaml, write a prod overlay that changes namespace, name prefix, replicas, and image tags, and apply it with kubectl apply -k.

Key Terminology

• Base — the directory of plain, valid Kubernetes YAML (chat-api Deployment, Service, PostgreSQL StatefulSet, etc.) that ships unchanged across environments and can be applied directly with kubectl apply -k base/.
• Overlay — a directory layered on top of a base that references it and adds environment-specific resources (e.g. a prod-only HPA) or modifies fields without editing the base files.
• Patch — a small YAML fragment (strategic merge or JSON 6902 patch) inside an overlay that changes specific fields like spec.replicas or a container's resource limits on a base resource.
• kustomization.yaml — the control file at the root of every base or overlay; lists resources:, patches:, images:, namespace:, namePrefix:, and other transformers Kustomize applies at build time.
• kubectl apply -k — the kubectl flag that runs kustomize build on a directory and applies the rendered manifests to the cluster in one step; use kubectl kustomize <dir> to preview without applying.

Concepts

Configuration as data, not as templates

Kustomize takes a fundamentally different approach than Helm: your base manifests are plain, valid Kubernetes YAML, and overlays apply transformations on top. You never see .Values.replicaCount syntax — you see replicas: 1, a real value that works in development. Anyone who can read YAML can read a Kustomize base without learning a templating language. The trade-off is that Kustomize is less flexible for conditional logic, but for "take these manifests and change these specific fields per environment" it is the right tool for the LLM chat stack.

One base, many overlays

A Kustomize project for the chat stack splits files into a shared base/ and one overlay per environment. The base/ holds every always-needed manifest — the chat API Deployment and Service, PostgreSQL and Redis StatefulSets and Services, the ConfigMap, the LimitRange, and the ResourceQuota — plus a kustomization.yaml that lists them. Each overlay (overlays/dev/, overlays/staging/, overlays/prod/) carries its own kustomization.yaml and patches; only overlays/prod/ adds an extra hpa.yaml because autoscaling is prod-only (see Code Walkthrough).

What an overlay can change

The five overlay mechanisms you'll use for the chat stack are: add extra resources: (the prod HPA), set namespace: so every resource lands in production, set namePrefix: (e.g. prod-) so resource names don't collide across environments in shared clusters, list patches: that strategic-merge into base fields like spec.replicas and container resources, and set images: to override image tags — this is what your CI/CD pipeline bumps on every release of llm-chat-api and gemini-proxy without editing base manifests.

Code Walkthrough

This walkthrough demonstrates the two concepts together — a plain-YAML base and a prod overlay that uses patches, name prefixing, namespace remap, and image-tag override. Start with the base.

The base — base/kustomization.yaml plus a real Deployment

Code snippetyaml
1# base/kustomization.yaml
2apiVersion: kustomize.config.k8s.io/v1beta1
3kind: Kustomization
4resources:
5  - chat-api-deployment.yaml
6  - chat-api-service.yaml
7  - postgresql-statefulset.yaml
8  - postgresql-service.yaml
9  - redis-statefulset.yaml
10  - redis-service.yaml
11  - configmap.yaml
12  - limitrange.yaml
13  - resourcequota.yaml
14commonLabels:
15  app.kubernetes.io/part-of: llm-chat
16---
17# base/chat-api-deployment.yaml
18apiVersion: apps/v1
19kind: Deployment
20metadata:
21  name: llm-chat-api
22  labels:
23    app: llm-chat-api
24spec:
25  replicas: 1
26  selector:
27    matchLabels:
28      app: llm-chat-api
29  template:
30    metadata:
31      labels:
32        app: llm-chat-api
33    spec:
34      containers:
35        - name: chat-api
36          image: gcr.io/my-project/llm-chat-api:latest
37          ports:
38            - containerPort: 8000
39          env:
40            - name: MODEL_NAME
41              valueFrom:
42                configMapKeyRef:
43                  name: llm-chat-config
44                  key: model-name
45          resources:
46            requests: { cpu: 100m, memory: 128Mi }
47            limits:   { cpu: 250m, memory: 256Mi }
48        - name: gemini-proxy
49          image: gcr.io/my-project/gemini-proxy:latest
50          ports:
51            - containerPort: 8081
52          resources:
53            requests: { cpu:  50m, memory:  64Mi }
54            limits:   { cpu: 100m, memory: 128Mi }

The resources: list names every base manifest Kustomize must include; commonLabels: is a transformer that automatically stamps app.kubernetes.io/part-of: llm-chat onto every resource. The Deployment beneath it is REAL YAML — replicas: 1, a working image reference, dev-sized resource requests and limits — so kubectl apply -k base/ gives a runnable dev deployment with the chat-api and the gemini-proxy sidecar, no overlay required.

The prod overlay — overlays/prod/kustomization.yaml

Code snippetyaml
1# overlays/prod/kustomization.yaml
2apiVersion: kustomize.config.k8s.io/v1beta1
3kind: Kustomization
4resources:
5  - ../../base
6  - hpa.yaml
7namespace: production
8namePrefix: prod-
9patches:
10  - path: replica-patch.yaml
11  - path: resource-patch.yaml
12images:
13  - name: gcr.io/my-project/llm-chat-api
14    newTag: v1.2.0
15  - name: gcr.io/my-project/gemini-proxy
16    newTag: v1.2.0

resources: pulls in the shared base via ../../base and adds the prod-only HPA. namespace: production and namePrefix: prod- rewrite every resource at build time (so llm-chat-api becomes prod-llm-chat-api in the production namespace). patches: strategic-merges replica and resource overrides onto the base Deployment. images: swaps the :latest tag in the base for the pinned release v1.2.0 on both the chat-api and the gemini-proxy — this is the one line CI bumps on every release. Apply with kubectl apply -k overlays/prod/, preview without applying via kubectl kustomize overlays/prod/, or diff against the live cluster with kubectl kustomize overlays/prod/ | kubectl diff -f -.

You'll know it works when kubectl kustomize overlays/prod/ prints rendered manifests whose every resource name starts with prod-, lives in namespace: production, references images tagged v1.2.0, and the chat-api Deployment shows the replica count from replica-patch.yaml instead of the base's 1.

Do's and Don'ts

Do's

1. ✓Do keep the base directly deployable — base manifests must run as-is with kubectl apply -k base/ against a dev cluster; if the base needs an overlay to be valid, the split is wrong.
2. ✓Do let images: in the overlay carry the release tag — CI bumps only the overlay's newTag, never the base manifest's image string, so every environment sees the release it asked for and the base never carries a moving :latest.
3. ✓Do preview with kubectl kustomize (or kubectl diff) before kubectl apply -k — render and read the merged output before pushing to prod; what you committed is not always what Kustomize produces after patches, prefixes, and namespace rewrites.

Don'ts

1. ✗Don't edit base manifests for a one-environment change — if only prod needs it, add a patch (or an extra resource) under overlays/prod/; mutating the base bleeds the change into every environment.
2. ✗Don't reach for the deprecated patchesStrategicMerge/patchesJson6902 fields — the modern patches: field handles both styles via auto-detection and is the only one that will keep working on current Kustomize versions.
3. ✗Don't share one overlay across environments via env-var conditionals — each environment gets its own overlays/<env>/ with its own kustomization.yaml; "if/else by env var" inside a single overlay is the templating sprawl Kustomize was built to avoid.