Design end-to-end architecture on GKE Autopilot

Introduction

When you deploy a GenAI pipeline as a tangle of ad-hoc services on GKE Autopilot, the first sign of trouble is usually a 2 AM page: a bursty ingestion job has starved the retrieval pods of CPU, queries are timing out, and nobody can name which service owns the contract that just broke. Teams that skip an explicit architecture step pay for it in cascading incidents, surprise Autopilot bills, and silent schema drift between stages. By the end of this lesson you'll be able to map a complete GenAI pipeline onto GKE Autopilot with isolated namespaces, per-stage resource budgets, and typed data-flow contracts between every component — so the system's shape is documented before any pod is scheduled.

Key Terminology

• ResourceQuota: A namespaced Kubernetes object that caps the aggregate CPU, memory, and pod count a namespace may consume, preventing one functional area from exhausting the cluster budget.
• LimitRange: A namespaced Kubernetes object that sets default resource requests and limits for containers that do not declare their own, giving Autopilot a predictable per-pod sizing signal.
• NamespaceProvisioner: The controller class shown in the walkthrough that declaratively creates a namespace together with its ResourceQuota and LimitRange so each pipeline stage launches with enforced isolation boundaries.

Concepts

A GKE Autopilot GenAI architecture is defined by three decisions made before any pod is scheduled. First, stage decomposition: the pipeline is split into ingestion, chunking, embedding, retrieval, and observability concerns, each owning its own namespace so that the blast radius of any single failure is bounded by design rather than by hope. Second, resource budgeting under per-pod billing: each namespace declares a ResourceQuota derived from its expected throughput — bursty and CPU-heavy for ingestion, steady-state low-latency for retrieval — and a LimitRange so unlabeled pods cannot accidentally request an oversized shape that quietly inflates the Autopilot bill. Third, contract-first integration: every inter-service edge in the data-flow graph is described by a typed message schema with explicit error and retry semantics, validated at process start so schema drift between stages fails loudly at deploy time rather than silently at 2 AM. The deliverable for this lesson is a single architecture document that captures all three — the namespace map, the per-stage resource budgets, and the contract for every arrow in the dataflow diagram.

Code Walkthrough

Namespace Isolation and Resource Planning

GKE Autopilot charges per pod based on actual resource requests, making namespace-level ResourceQuota and LimitRange definitions critical for cost control. Each functional area gets its own namespace with explicit CPU and memory budgets. The architecture separates ingestion workloads (bursty, CPU-heavy during PDF processing) from retrieval workloads (latency-sensitive, memory-heavy for vector operations) so that a sudden batch upload cannot starve query-serving pods of resources.

NamespaceProvisioner automates the creation of isolated namespaces with resource quotas, limit ranges, and network policies. The class reads a declarative configuration and applies Kubernetes resources that enforce per-namespace boundaries.

1from kubernetes import client, config
2
3class NamespaceProvisioner:
4    def __init__(self):
5        config.load_incluster_config()
6        self.core = client.CoreV1Api()
7        self.net = client.NetworkingV1Api()
8
9    def provision(
10        self, name: str, cpu: str, mem: str,
11    ) -> None:
12        ns = client.V1Namespace(
13            metadata=client.V1ObjectMeta(
14                name=name,
15                labels={
16                    "app.kubernetes.io/part-of":
17                        "genai-pipeline",
18                },
19            ),
20        )
21        self.core.create_namespace(body=ns)
22        quota = client.V1ResourceQuota(
23            metadata=client.V1ObjectMeta(
24                name=f"{name}-quota",
25                namespace=name,
26            ),
27            spec=client.V1ResourceQuotaSpec(
28                hard={
29                    "requests.cpu": cpu,
30                    "requests.memory": mem,
31                    "pods": "50",
32                },
33            ),
34        )
35        self.core.create_namespaced_resource_quota(
36            namespace=name, body=quota,
37        )
38        limit = client.V1LimitRange(
39            metadata=client.V1ObjectMeta(
40                name=f"{name}-limits",
41                namespace=name,
42            ),
43            spec=client.V1LimitRangeSpec(
44                limits=[
45                    client.V1LimitRangeItem(
46                        type="Container",
47                        default={"cpu": "500m",
48                                 "memory": "512Mi"},
49                        default_request={
50                            "cpu": "250m",
51                            "memory": "256Mi"},
52                    ),
53                ],
54            ),
55        )
56        self.core.create_namespaced_limit_range(
57            namespace=name, body=limit,
58        )

Data Flow Contracts

Each inter-service boundary needs a formal contract specifying the input schema, output schema, error handling protocol, and retry semantics. Without these contracts, upstream changes silently break downstream consumers. The architecture document captures these as typed interfaces that both producer and consumer services validate at startup.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Give each pipeline stage its own namespace with an explicit ResourceQuota and LimitRange so Autopilot's per-pod billing is bounded per stage and bursty ingestion cannot starve latency-sensitive retrieval pods.
2. ✓Define a typed data-flow contract — input schema, output schema, error codes, retry policy — for every service-to-service edge in the diagram, and validate it at process startup so a producer change cannot silently break a downstream consumer.
3. ✓Treat the architecture document (namespace map + budgets + contracts) as the gating artifact for the first deploy: no pod ships until its stage appears in the map and its contracts are reviewed.

Don'ts

1. ✗Don't co-locate ingestion, embedding, and retrieval workloads in a single default namespace — without quota isolation a single batch upload can exhaust the cluster budget under Autopilot's per-pod pricing.
2. ✗Don't rely on informal "we agreed on a JSON shape" handoffs between stages; absent a validated contract, schema drift surfaces as 2 AM pages instead of deploy-time failures.
3. ✗Don't pick CPU/memory requests by guesswork — derive each namespace's quota from the stage's measured throughput profile (bursty vs. steady) so Autopilot sizing matches the real workload.