Deploy data services on Kubernetes with StatefulSets

Introduction

When you deploy a database on Kubernetes using a standard Deployment, the first pod reschedule silently destroys all stored data and leaves replicas unable to locate each other. PostgreSQL, Redis, and Kafka each require stable pod identities, dedicated persistent storage, and deterministic DNS names — none of which Deployments provide. StatefulSets with volumeClaimTemplates solve all three: each pod gets its own PersistentVolumeClaim, a stable hostname through a headless Service, and the ordered startup sequence that replication protocols depend on.

By the end of this lesson, you will be able to write StatefulSet manifests for all three services, apply the correct storage sizing for GenAI workloads, and deploy them programmatically using the Kubernetes Python client.

Key Terminology

• StatefulSet — A Kubernetes workload controller that manages pods with stable identities, ordered scaling, and persistent storage bindings, designed for databases and message brokers; the only correct primitive for data services on Kubernetes.
• PersistentVolumeClaim (PVC) — A request for storage that abstracts the underlying provisioner (GCE Persistent Disk, AWS EBS, or local NVMe), enabling portable storage declarations; the per-pod handle that survives pod rescheduling.
• Headless Service — A Service with clusterIP: None that creates individual DNS A-records per pod instead of load-balancing, required by StatefulSets for peer discovery during replication and quorum elections.
• volumeClaimTemplate — A template inside a StatefulSet spec that automatically creates one PVC per pod replica, ensuring each pod gets its own dedicated storage volume bound by ordinal index.
• StorageClass — A Kubernetes object that defines the provisioner, parameters, and reclaim policy for dynamically provisioned PersistentVolumes; only classes with allowVolumeExpansion: true allow you to grow a PVC later without a rebuild.
• WAL (Write-Ahead Log) — PostgreSQL's transaction journal that records changes before they are applied to data files, enabling crash recovery and point-in-time restore; losing WAL means losing the last seconds of writes.

Concepts

The diagram below shows how a StatefulSet wires together a headless Service, ordinal pods, and per-pod PVCs to produce stable identities and dedicated storage.

Storage Sizing and Access Mode Considerations

Choosing the wrong storage size or access mode is a deployment-time decision that is expensive to fix later. PVC expansion is supported on most cloud providers, but only for StorageClasses with allowVolumeExpansion: true, and expanding requires a pod restart. Shrinking a PVC is never supported — you must create a new PVC and migrate data.

Follow these sizing guidelines for GenAI workloads:

• PostgreSQL: Allocate 10Gi minimum for development, 100Gi+ for production with embedding tables. A single pgvector index on 1M 1536-dimensional embeddings consumes approximately 6Gi. Use ReadWriteOnce access mode — a primary database should never be mounted by multiple nodes simultaneously.
• Redis: Allocate 2Gi for caching-only workloads (AOF compaction keeps disk usage low). For Redis as a primary vector store using RedisSearch, allocate memory equal to your dataset size plus 30% overhead. Use ReadWriteOnce.
• Kafka: Allocate 50Gi per broker as a starting point, then scale based on retention.bytes and throughput. With a 7-day retention at 10MB/s ingest, each broker needs approximately 600Gi. Use ReadWriteOnce — Kafka handles replication at the application level, not the storage level.

Common Failure Modes and Mitigations

When a StatefulSet deployment stalls, the root cause falls into one of three categories. First, volume provisioning failures: the StorageClass does not exist, the cloud provider quota is exhausted, or the availability zone has no capacity. Check PVC events with kubectl describe pvc data-postgres-0 and look for ProvisioningFailed events. Second, image pull failures: the container registry requires authentication that is not configured as an imagePullSecret on the pod spec. Third, readiness probe failures: the pod starts but the database process inside crashes during initialization — typically caused by incorrect PGDATA paths, missing secrets, or insufficient memory limits triggering OOMKill.

For each failure, the health check function below reports the pod stuck in Pending or CrashLoopBackOff, giving your CI/CD pipeline a programmatic signal to halt the deployment and alert the on-call engineer rather than proceeding with downstream service deployments that depend on these data stores being healthy.

Code Walkthrough

Building on the storage sizing guidelines and access-mode constraints from the Concepts section, the following Python script translates those rules directly into a working PostgreSQL StatefulSet: 10 Gi of ReadWriteOnce storage, a headless Service for stable per-pod DNS, and a volumeClaimTemplate that provisions one dedicated PVC per ordinal replica.

Code snippetpython
1from kubernetes import client, config
2
3def deploy_postgres(
4    namespace: str = "data",
5    storage_class: str = "standard",
6    pg_image: str = "postgres:16.2",
7) -> None:
8    config.load_kube_config()
9    apps_v1 = client.AppsV1Api()
10    core_v1 = client.CoreV1Api()
11
12    # Headless Service — cluster_ip="None" creates per-pod DNS A-records
13    svc = client.V1Service(
14        metadata=client.V1ObjectMeta(name="postgres-svc", namespace=namespace),
15        spec=client.V1ServiceSpec(
16            cluster_ip="None",
17            selector={"app": "postgres"},
18            ports=[client.V1ServicePort(port=5432, name="pg")],
19        ),
20    )
21    core_v1.create_namespaced_service(namespace=namespace, body=svc)
22
23    container = client.V1Container(
24        name="postgres",
25        image=pg_image,
26        ports=[client.V1ContainerPort(container_port=5432)],
27        env=[
28            client.V1EnvVar(name="POSTGRES_DB", value="genai"),
29            client.V1EnvVar(name="POSTGRES_PASSWORD", value="changeme"),
30            client.V1EnvVar(name="PGDATA", value="/var/lib/postgresql/data/pgdata"),
31        ],
32        volume_mounts=[
33            client.V1VolumeMount(name="data", mount_path="/var/lib/postgresql/data")
34        ],
35    )
36
37    pvc_template = client.V1PersistentVolumeClaim(
38        metadata=client.V1ObjectMeta(name="data"),
39        spec=client.V1PersistentVolumeClaimSpec(
40            access_modes=["ReadWriteOnce"],
41            storage_class_name=storage_class,
42            resources=client.V1ResourceRequirements(
43                requests={"storage": "10Gi"}
44            ),
45        ),
46    )
47
48    statefulset = client.V1StatefulSet(
49        metadata=client.V1ObjectMeta(name="postgres", namespace=namespace),
50        spec=client.V1StatefulSetSpec(
51            service_name="postgres-svc",
52            replicas=1,
53            selector=client.V1LabelSelector(match_labels={"app": "postgres"}),
54            template=client.V1PodTemplateSpec(
55                metadata=client.V1ObjectMeta(labels={"app": "postgres"}),
56                spec=client.V1PodSpec(containers=[container]),
57            ),
58            volume_claim_templates=[pvc_template],
59        ),
60    )
61
62    apps_v1.create_namespaced_stateful_set(namespace=namespace, body=statefulset)
63    print("PostgreSQL StatefulSet created.")

Several details enforce the sizing and identity rules from the Concepts section. Setting cluster_ip="None" on the Service is what makes it headless — the cluster DNS plane creates individual A-records (postgres-0.postgres-svc.data.svc.cluster.local) rather than a single load-balanced virtual IP, which replication protocols require for direct peer addressing. The volume_claim_templates list holds one template named data; Kubernetes appends the pod ordinal to produce PVC names data-postgres-0, data-postgres-1, and so on. The ReadWriteOnce access mode ensures no two nodes can mount the same volume simultaneously, protecting WAL integrity on a single-primary setup.

For Redis and Kafka, the same StatefulSet structure applies with the storage sizes from the Concepts section substituted in: Redis requests 2Gi (sufficient for AOF-backed caching workloads), and each Kafka broker requests 50Gi. The ordered creation guarantee is automatic — kafka-1 will not start until kafka-0 is Ready, satisfying the controller election sequence.

After calling deploy_postgres(), run the following readiness check before proceeding:

Code snippetpython
1import time
2from kubernetes import client, config
3
4def wait_for_statefulset(name: str, namespace: str = "data", timeout: int = 120) -> bool:
5    config.load_kube_config()
6    apps_v1 = client.AppsV1Api()
7    deadline = time.time() + timeout
8    while time.time() < deadline:
9        sts = apps_v1.read_namespaced_stateful_set(name=name, namespace=namespace)
10        if sts.status.ready_replicas == sts.spec.replicas:
11            return True
12        time.sleep(5)
13    return False
14
15ready = wait_for_statefulset("postgres")
16print("Ready:" if ready else "Timed out waiting for", "postgres")

Verify by confirming wait_for_statefulset("postgres") returns True and that kubectl get statefulsets,pvc -n data shows the StatefulSet at 1/1 ready replicas and the PVC data-postgres-0 in Bound state.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do set cluster_ip="None" on the Service linked to every StatefulSet — this makes the Service headless, causing the cluster DNS plane to create per-pod A-records like postgres-0.postgres-svc.data.svc.cluster.local instead of a single virtual IP, which replication protocols in PostgreSQL, Redis, and Kafka require for direct peer addressing.
2. ✓Do size volumeClaimTemplates storage per workload role — PostgreSQL needs 10Gi for WAL and data files, Redis needs 2Gi for AOF persistence, and Kafka brokers need 50Gi per broker for log segments; under-provisioning any of these causes write failures under GenAI workload volumes without a clear eviction warning.
3. ✓Do poll sts.status.ready_replicas == sts.spec.replicas with wait_for_statefulset() before proceeding to dependent services — StatefulSets create pods sequentially (kafka-1 will not start until kafka-0 is Ready), so driving subsequent deploys off a timed sleep instead of a readiness check races against the ordered startup guarantee and can leave broker election or replication initialization incomplete.

Don'ts

1. ✗Don't use a Deployment instead of a StatefulSet to run PostgreSQL, Redis, or Kafka — Deployments provide no stable pod identity, no per-pod PVCs, and no ordered startup, so the first pod reschedule silently destroys stored data and leaves replicas unable to locate each other via DNS.
2. ✗Don't omit service_name from V1StatefulSetSpec or mismatch it with the headless Service's metadata.name — the service_name field is what binds the StatefulSet to its headless Service and governs the <pod>.<svc>.<namespace>.svc.cluster.local DNS template; a mismatch means pods are unreachable by stable hostname even though they start successfully.
3. ✗Don't specify access_modes=["ReadWriteMany"] for single-primary PostgreSQL — ReadWriteOnce is required so no two nodes can mount the same volume simultaneously, protecting WAL integrity; using ReadWriteMany on a single-primary setup allows concurrent mounts that can corrupt the write-ahead log without raising an immediate error.