Deploy multi-stage safety pipeline on GKE

Introduction

When you build a content safety system as a single service, a surge in classifier traffic can exhaust its resources and take down moderation entirely—blocking users or, worse, letting harmful content through unchecked. Splitting each concern into its own independently deployed stage removes that single point of failure. By the end of this lesson, you will be able to generate the Dockerfiles, Kubernetes Deployment manifests, Helm values files, and Horizontal Pod Autoscaler configurations that run each pipeline stage—intake, classifier, aggregator, and action—as an independently scalable microservice on GKE connected through Redis.

Key Terminology

• Stage isolation — the design pattern of running each pipeline phase (intake, classifier, aggregator, action) as a separate, independently deployed microservice so that a surge or crash in one stage cannot exhaust shared resources or take down the entire pipeline.
• HorizontalPodAutoscaler (HPA) — a Kubernetes autoscaling/v2 resource, produced by generate_hpa_manifest, that monitors a Deployment's CPU utilization and automatically adjusts its replica count between minReplicas and maxReplicas to match traffic load.
• Readiness probe — a periodic HTTP check against /health with initialDelaySeconds: 5 that gates traffic routing; Kubernetes withholds requests from a pod until this probe succeeds, preventing a stage from receiving traffic before it has finished initializing.
• Liveness probe — a periodic HTTP check against /health with initialDelaySeconds: 15 that detects a container that started successfully but later hung or deadlocked, triggering an automatic restart without human intervention.
• Resource requests and limits — per-container CPU and memory bounds declared in generate_deployment_manifest and surfaced in generate_helm_values; requests tell the Kubernetes scheduler how much capacity to reserve on a node, while limits cap what a container can consume during a burst, preventing one busy stage from starving its neighbors.
• Helm values — the nested configuration dictionary produced by generate_helm_values that records each stage's replica count, resource profile, and autoscaling parameters in a single structure consumed by Helm chart templates to render per-stage Deployments and HPAs.

Concepts

Stage Isolation as a Safety Property

A content safety pipeline that runs as a single service has a hidden vulnerability: any component—a CPU-heavy regex scan, a slow model inference call, a memory-hungry toxicity classifier—can exhaust the shared process and block all moderation decisions simultaneously. Decomposing the pipeline into four independent stages—intake, classifier, aggregator, and action—removes that coupling. Each stage is its own Deployment, scheduled on its own pods with its own CPU and memory quota. A classifier spike no longer interferes with the action stage, and a crash in aggregation does not orphan requests still queued in intake.

Redis connects the stages as a lightweight message broker so each reads from and writes to named queues rather than calling its neighbors directly. This decouples stage lifecycles: you can restart the classifier independently, roll a new aggregation algorithm without touching intake pods, and size each stage's replica count to its own throughput characteristics—not the worst-case load of the busiest sibling.

Per-Stage Resource Profiles

Not every stage consumes compute equally. The classifier stage runs regex scans and may load an ML toxicity model, making it CPU and memory intensive. The intake, aggregator, and action stages are lightweight by comparison. generate_deployment_manifest therefore accepts explicit cpu_request, memory_request, cpu_limit, and memory_limit per stage rather than applying a single shared profile. Resource requests tell the Kubernetes scheduler which node has sufficient capacity for a pod; resource limits cap what each container can consume during a burst, preventing one busy stage from starving its co-located neighbors (see Code Walkthrough for how each parameter is threaded into the manifest).

Health Probes and Traffic Gating

Both probes in generate_deployment_manifest target /health on port 8080, but they fire at different times and serve distinct purposes. The readiness probe (initialDelaySeconds: 5) gates traffic routing: until it passes, the pod is removed from the Service's endpoint list and receives no requests. The liveness probe (initialDelaySeconds: 15) detects a container that started successfully but later hung or deadlocked, triggering an automatic restart. The staggered delays are intentional—the liveness probe waits long enough for the readiness probe to have already succeeded, so a slow-starting container is not killed before it has had a chance to initialize.

Per-Stage Autoscaling with HPA and Helm

generate_hpa_manifest produces an autoscaling/v2 HorizontalPodAutoscaler that watches a specific Deployment and scales its replica count when average CPU utilization crosses targetCPU. Because each stage gets its own HPA, the classifier can scale to ten replicas during a content moderation spike while the action stage holds steady at two—right-sizing compute to actual demand rather than over-provisioning every stage to survive the peak load of the busiest one. generate_helm_values collects the replica counts, resource bounds, and autoscaling parameters for all stages into one nested dictionary that Helm chart templates consume, keeping per-stage configuration centralized and eliminating the need to edit individual manifest files when tuning thresholds.

Code Walkthrough

Building on the four-stage pipeline architecture—intake, classifier, aggregator, and action—you will now produce the container and Kubernetes artifacts that deploy each stage as an independent GKE microservice.

The first pair of functions handles per-stage container and Deployment generation. generate_dockerfile builds a Python 3.11-slim image for any named stage: it copies a requirements.txt, installs dependencies without cache, exposes port 8080, attaches a health check that polls the /health endpoint, and starts uvicorn. generate_deployment_manifest returns a Kubernetes Deployment dictionary carrying the stage name, replica count, resource requests and limits, and both readiness and liveness probes targeting the same /health path. The readiness probe begins after five seconds to gate traffic routing; the liveness probe begins after fifteen seconds to detect hung processes.

Code snippetpython
1def generate_dockerfile(stage_name: str, requirements: list[str]) -> str:
2    requirements_str = "\n".join(requirements)
3    return f"""FROM python:3.11-slim
4WORKDIR /app
5COPY requirements.txt .
6RUN pip install --no-cache-dir -r requirements.txt
7COPY {stage_name}/ .
8EXPOSE 8080
9HEALTHCHECK --interval=30s --timeout=5s \\
10  CMD python -c "import httpx; httpx.get('http://localhost:8080/health')"
11CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]
12"""
13
14def generate_deployment_manifest(
15    stage_name: str,
16    replicas: int,
17    cpu_request: str,
18    memory_request: str,
19    cpu_limit: str,
20    memory_limit: str,
21) -> dict:
22    return {
23        "apiVersion": "apps/v1",
24        "kind": "Deployment",
25        "metadata": {
26            "name": f"safety-{stage_name}",
27            "labels": {"app": f"safety-{stage_name}", "tier": "safety-pipeline"},
28        },
29        "spec": {
30            "replicas": replicas,
31            "selector": {"matchLabels": {"app": f"safety-{stage_name}"}},
32            "template": {
33                "metadata": {"labels": {"app": f"safety-{stage_name}"}},
34                "spec": {
35                    "containers": [{
36                        "name": stage_name,
37                        "image": f"gcr.io/PROJECT/safety-{stage_name}:latest",
38                        "ports": [{"containerPort": 8080}],
39                        "resources": {
40                            "requests": {"cpu": cpu_request, "memory": memory_request},
41                            "limits": {"cpu": cpu_limit, "memory": memory_limit},
42                        },
43                        "readinessProbe": {
44                            "httpGet": {"path": "/health", "port": 8080},
45                            "initialDelaySeconds": 5,
46                            "periodSeconds": 10,
47                        },
48                        "livenessProbe": {
49                            "httpGet": {"path": "/health", "port": 8080},
50                            "initialDelaySeconds": 15,
51                            "periodSeconds": 20,
52                        },
53                    }],
54                },
55            },
56        },
57    }

Resource profiles differ meaningfully by stage. The classifier stage is CPU-intensive for regex scanning and may be memory-intensive if it loads an ML toxicity model. The aggregator, intake, and action stages are lightweight by comparison. Configuring resource requests and limits independently per stage ensures the scheduler places pods on nodes with sufficient capacity and prevents one busy stage from starving the others.

The second pair of functions handles Helm values and autoscaling. generate_helm_values iterates over a list of stage configuration dictionaries and builds a nested values structure consumed by the Helm chart templates—each entry records replica count, resource requests and limits, and autoscaling parameters including minimum replicas, maximum replicas, and CPU utilization target. generate_hpa_manifest produces the HorizontalPodAutoscaler resource that wires those parameters to an autoscaling/v2 object pointing at the stage's Deployment, allowing GKE to scale each stage horizontally as traffic varies.

Code snippetpython
1def generate_helm_values(stages: list[dict]) -> dict:
2    values = {"stages": {}}
3    for stage in stages:
4        values["stages"][stage["name"]] = {
5            "replicas": stage.get("replicas", 2),
6            "resources": {
7                "requests": {
8                    "cpu": stage.get("cpu_request", "100m"),
9                    "memory": stage.get("memory_request", "128Mi"),
10                },
11                "limits": {
12                    "cpu": stage.get("cpu_limit", "500m"),
13                    "memory": stage.get("memory_limit", "512Mi"),
14                },
15            },
16            "autoscaling": {
17                "enabled": stage.get("autoscaling", True),
18                "minReplicas": stage.get("min_replicas", 2),
19                "maxReplicas": stage.get("max_replicas", 10),
20                "targetCPU": stage.get("target_cpu", 70),
21            },
22        }
23    return values
24
25def generate_hpa_manifest(
26    stage_name: str, min_replicas: int, max_replicas: int, target_cpu: int
27) -> dict:
28    return {
29        "apiVersion": "autoscaling/v2",
30        "kind": "HorizontalPodAutoscaler",
31        "metadata": {"name": f"safety-{stage_name}-hpa"},
32        "spec": {
33            "scaleTargetRef": {
34                "apiVersion": "apps/v1",
35                "kind": "Deployment",
36                "name": f"safety-{stage_name}",
37            },
38            "minReplicas": min_replicas,
39            "maxReplicas": max_replicas,
40            "metrics": [{
41                "type": "Resource",
42                "resource": {
43                    "name": "cpu",
44                    "target": {"type": "Utilization", "averageUtilization": target_cpu},
45                },
46            }],
47        },
48    }

Confirm that calling generate_deployment_manifest("classifier", 2, "500m", "512Mi", "2000m", "2Gi") returns a dictionary whose kind is "Deployment" and metadata["name"] is "safety-classifier", and that calling generate_hpa_manifest("classifier", 2, 10, 70) returns a dictionary whose kind is "HorizontalPodAutoscaler" with spec["scaleTargetRef"]["name"] matching that same Deployment name.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do set readinessProbe initialDelaySeconds to 5 and livenessProbe initialDelaySeconds to 15 on every stage's Deployment — the staggered delay ensures traffic is only routed to a pod after its /health endpoint is responsive, while the longer liveness window gives uvicorn time to fully initialize before GKE begins killing hung processes.
2. ✓Do configure resource requests and limits independently per stage based on its actual workload profile — the classifier stage warrants higher CPU and memory (e.g., 500m/512Mi request, 2000m/2Gi limit) to accommodate regex scanning and optional ML model loading, while intake, aggregator, and action stages can use lighter profiles; mixing these up causes the scheduler to misplace pods or lets one busy stage starve the others.
3. ✓Do wire each stage's generate_hpa_manifest output to the autoscaling/v2 API with per-stage minReplicas, maxReplicas, and targetCPU values sourced from generate_helm_values — this keeps the HPA's scaleTargetRef name (safety-<stage>) consistent with the Deployment's metadata.name, so GKE can scale each pipeline stage independently as classifier traffic spikes without affecting intake or action pods.

Don'ts

1. ✗Don't share a single Deployment or resource profile across all four pipeline stages — collapsing intake, classifier, aggregator, and action into one service re-introduces the single point of failure the multi-stage architecture is designed to eliminate; a classifier traffic surge will again exhaust resources and block or bypass moderation entirely.
2. ✗Don't omit the HEALTHCHECK instruction from generate_dockerfile or remove the /health path from the probe definitions in generate_deployment_manifest — without the httpGet probes polling /health on port 8080, Kubernetes cannot distinguish a hung uvicorn process from a healthy one, and a crashed stage will continue to receive traffic until its pod is eventually evicted.
3. ✗Don't hardcode autoscaling parameters (minReplicas, maxReplicas, targetCPU) directly into generate_hpa_manifest calls without passing them through generate_helm_values first — bypassing the Helm values layer breaks the single source of truth for autoscaling configuration, causing the HPA and the chart templates to drift out of sync when parameters are tuned for production traffic.