Deploy platform control plane with Helm and ArgoCD

Introduction

Engineers often manage the control plane the same way they manage one-off fixes — kubectl apply from a laptop, no audit history, no rollback path, no version control. When that control plane provisions your gateways, registries, and agent runtimes, a bad apply can cascade into a full platform outage with no clear path back. This lesson teaches you to package the control plane as a Helm chart, declare its desired state as an ArgoCD Application manifest, and enforce a GitOps workflow where every change is a reviewed commit and every rollback is a git revert.

Key Terminology

• GitOps — A deployment discipline that uses Git as the single source of truth for cluster state; every change to the control plane is a commit reviewed in a PR, and every rollback is a git revert — no out-of-band kubectl apply is ever the authoritative change path.
• ArgoCD Application — A Kubernetes custom resource (argoproj.io/v1alpha1) that binds a Helm chart at a specific Git targetRevision to a destination namespace and drives the continuous reconciliation loop that keeps the cluster in sync with Git.
• selfHeal — The syncPolicy.automated.selfHeal: true flag in the ArgoCD Application manifest; it detects drift caused by any manual kubectl edit and reverts the cluster back to the Git-committed state within seconds, making Git the only valid change path.
• prune — The syncPolicy.automated.prune: true flag that instructs ArgoCD to delete cluster resources removed from the chart in Git, preventing orphaned Deployments and Services from accumulating as ghosts after a resource is retired.
• sync wave — A resource-ordering mechanism controlled by the annotation argocd.argoproj.io/sync-wave; setting a CRD template to wave "-5" guarantees the schema is registered in the API server before any dependent resource is applied, avoiding schema-not-found failures on first rollout.
• image pinning — The practice of setting a fixed content digest or version tag in values-production.yaml so that {{ .Values.image.tag | default .Chart.AppVersion }} resolves to a deterministic image, preventing silent behavior changes from mutable tags like :latest.

Concepts

The Control Plane Deserves the Same Rigor It Enforces

The platform control plane provisions everything else — gateways, service registries, agent runtimes. That dependency relationship creates an asymmetric blast radius: a bad kubectl apply to the control plane is not a single service outage, it is a potential loss of every workload the platform manages. Yet teams routinely deploy the control plane the same way they handle one-off fixes — directly from a developer's laptop, with no audit trail, no peer review, and no rollback path shorter than manual reconstruction.

Packaging the control plane as a Helm chart and placing it under ArgoCD management applies the same deployment discipline the platform imposes on the services it runs. The chart makes the control plane's manifests versioned, parameterizable, and testable with helm template | kubectl apply --dry-run=server before anything touches the cluster.

How Helm Structures Environment Variance

A Helm chart separates what is structural from what varies by environment. The templates/ directory holds the Deployment, Service, and RBAC resources — these are identical across staging and production. What changes lives in values-production.yaml and values-staging.yaml: replica counts, resource limits, and critically, the image tag.

The expression {{ .Values.image.tag | default .Chart.AppVersion }} is the image-pinning hook (see Code Walkthrough). Production values set a pinned digest; staging may omit the field and fall back to the chart's appVersion. Credentials are never stored in values files — envFrom: secretRef delegates credential injection to external-secrets-operator, keeping secrets out of Git while remaining fully declarative.

The readinessProbe and livenessProbe serve distinct purposes that must not be conflated: the readiness probe gates whether the pod receives traffic; the liveness probe triggers a pod restart on a hard hang. Routing traffic before the readiness probe passes drops requests silently; restarting a pod that is merely slow to initialize produces a hot-restart loop.

ArgoCD's Reconciliation Loop and Drift Enforcement

ArgoCD runs a continuous reconciliation loop: it renders the chart against the target valueFiles, diffs the result against live cluster state, and drives any divergence to zero. Three flags in syncPolicy define how aggressively that enforcement operates.

selfHeal: true means any manual kubectl edit is reversed within seconds — Git is the only valid change path. prune: true means resources deleted from the chart are removed from the cluster; without it, retired Deployments and Services accumulate silently. ApplyOutOfSyncOnly limits each reconciliation pass to only the drifted resources, keeping syncs fast as the chart grows.

When the chart ships CRDs alongside application resources, installation order matters. ArgoCD processes resources in waves defined by the argocd.argoproj.io/sync-wave annotation. Annotating a CRD template with sync-wave: "-5" guarantees the schema exists in the API server before any dependent resource is applied — otherwise the API Deployment starts against a schema that does not yet exist and fails immediately. The two canonical health signals — Health: Healthy and Sync Status: Synced from argocd app get platform-control-plane — together confirm that live cluster state matches the Git commit exactly and no drift is pending.

Code Walkthrough

Building on the Helm + ArgoCD discipline from the Concepts section, the two artifacts that put it into practice are the Helm deployment template and the ArgoCD Application manifest.

The Helm Deployment Template

The control-plane chart follows a standard layout — Chart.yaml, per-environment values-*.yaml overrides, and a templates/ directory for each K8s resource. The deployment template wires those values to live cluster state:

Code snippetyaml
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: {{ include "platform.fullname" . }}-api
5  labels: {{- include "platform.labels" . | nindent 4 }}
6spec:
7  replicas: {{ .Values.api.replicaCount }}
8  selector:
9    matchLabels: {{- include "platform.selectorLabels" . | nindent 6 }}
10      app.kubernetes.io/component: api
11  template:
12    metadata:
13      labels: {{- include "platform.selectorLabels" . | nindent 8 }}
14        app.kubernetes.io/component: api
15    spec:
16      serviceAccountName: {{ include "platform.fullname" . }}
17      containers:
18      - name: api
19        image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
20        ports:
21        - name: http
22          containerPort: 8000
23        envFrom:
24        - secretRef: {name: {{ include "platform.fullname" . }}-db}
25        readinessProbe:
26          httpGet: {path: /healthz/ready, port: http}
27          initialDelaySeconds: 5
28          periodSeconds: 5
29        livenessProbe:
30          httpGet: {path: /healthz/live, port: http}
31          initialDelaySeconds: 30
32          periodSeconds: 30
33        resources: {{- toYaml .Values.api.resources | nindent 10 }}

{{ .Values.image.tag | default .Chart.AppVersion }} is the image-pinning hook: production values-production.yaml sets a pinned digest; staging may leave the field unset and fall back to the chart's appVersion. envFrom: secretRef pulls database credentials from a secret managed by external-secrets-operator — credentials never appear in Helm values. readinessProbe gates traffic; livenessProbe restarts the pod on hard failure. Confusing the two causes either hot-restart loops or silent traffic to a non-ready pod.

The ArgoCD Application Manifest

The ArgoCD Application resource binds the chart to a cluster namespace and enforces the sync policy:

Code snippetyaml
1apiVersion: argoproj.io/v1alpha1
2kind: Application
3metadata:
4  name: platform-control-plane
5  namespace: argocd
6spec:
7  project: platform
8  source:
9    repoURL: https://github.com/your-org/platform-config
10    targetRevision: main
11    path: charts/platform-control-plane
12    helm:
13      valueFiles:
14        - values-production.yaml
15  destination:
16    server: https://kubernetes.default.svc
17    namespace: platform-system
18  syncPolicy:
19    automated:
20      prune: true
21      selfHeal: true
22    syncOptions:
23      - CreateNamespace=true
24      - ApplyOutOfSyncOnly=true
25    retry:
26      limit: 5
27      backoff:
28        duration: 5s
29        maxDuration: 3m
30        factor: 2

selfHeal: true is the GitOps enforcement mechanism — any manual kubectl edit is reverted within seconds, making Git the only valid change path. prune: true ensures resources deleted from Git are removed from the cluster; without it, old deployments and services accumulate as ghosts. ApplyOutOfSyncOnly limits each reconciliation to only the drifted resources, keeping syncs fast as the chart grows. For charts that ship CRDs alongside the API deployment, annotating the CRD template with argocd.argoproj.io/sync-wave: "-5" ensures CRDs install before dependent resources — otherwise the API pod starts against a schema that does not yet exist.

Confirm that argocd app get platform-control-plane reports Health: Healthy and Sync Status: Synced with an empty diff — those two states together mean the live cluster matches the Git commit exactly and no out-of-band changes are pending.

Do's and Don'ts

Having walked through the Helm chart template and ArgoCD Application manifest above, the following do's and don'ts capture the operational discipline that keeps the control-plane rollout safe in practice.

Do's

1. ✓Do pin the image tag to a digest in values-production.yaml — leaving .Values.image.tag unset falls back to .Chart.AppVersion, which is a mutable tag that can silently pull a different image on pod restart; a pinned digest guarantees the running image is exactly what was reviewed and merged.
2. ✓Do set selfHeal: true and prune: true together in the ArgoCD Application syncPolicy — selfHeal reverts any manual kubectl edit within seconds (making Git the only valid change path), while prune removes resources deleted from Git so old deployments and services don't accumulate as unreachable ghosts.
3. ✓Do annotate CRD templates with argocd.argoproj.io/sync-wave: "-5" when the chart ships CRDs alongside the API deployment — sync waves guarantee CRDs are installed before dependent resources, preventing the control-plane API pod from starting against a schema that does not yet exist in the cluster.

Don'ts

1. ✗Don't store database credentials in Helm values files — envFrom: secretRef pulling from external-secrets-operator is the correct pattern; credentials in values-production.yaml end up in Git history and in the ArgoCD UI, exposing them to anyone with repo or dashboard read access.
2. ✗Don't confuse readinessProbe (/healthz/ready) with livenessProbe (/healthz/live) — wiring them to the same endpoint or swapping their roles either causes hot-restart loops (liveness fires before the app is ready) or silently routes traffic to a non-ready pod (readiness never gates the Service endpoints).
3. ✗Don't omit ApplyOutOfSyncOnly from syncOptions as the chart grows — without it, every reconciliation re-applies every resource in the chart regardless of drift state, making syncs progressively slower and increasing the blast radius of a bad reconciliation as the number of control-plane resources increases.