Deploy the LLM app as your first Kubernetes pod

Introduction

When you've spent Chapter 1 building an LLM container image, the next question is brutally practical: how do you actually get it running on the cluster, and how do you know it's healthy enough to take a request? A pod that's stuck in Pending for hours, or one that flips into CrashLoopBackOff the moment the kubelet pulls the image, will silently break every downstream demo until you can read the lifecycle correctly. By the end of this lesson you'll be able to write a pod manifest for your Chapter 1 LLM image, apply it to the cluster, watch the lifecycle transitions in real time, and confirm the inference endpoint responds via port forwarding.

Key Terminology

• Pod — the smallest deployable object in Kubernetes: a wrapper around one or more containers sharing a network namespace, an IP, and volumes. It is what you actually deploy to run the LLM image from Chapter 1.
• Pod phase — the high-level state the API server reports: Pending, Running, Succeeded, Failed, Unknown. Where a pod is stuck tells you which subsystem is the failure source.
• CrashLoopBackOff — the waiting reason the kubelet attaches when a container keeps exiting; restarts are throttled with exponential backoff (10s, 20s, 40s, …). Almost always points to a bad command, missing env var, or unmounted weights path inside your LLM container.
• ImagePullBackOff — the waiting reason when the kubelet cannot pull your image. For GenAI workloads this usually means a typo in the tag from Chapter 1 or missing registry credentials.
• Port forwarding — kubectl port-forward opens a TCP tunnel from your laptop to the pod's container port, bypassing services and load balancers. It is the fastest way to confirm your LLM pod answers /generate before you wire anything else up.

Concepts

Pods wrap your Chapter 1 container

A pod is not a container; it is the Kubernetes abstraction around it. When you write a manifest you tell the API server which image to run, what resources it needs (including GPUs for the LLM), what environment variables to inject, and which volumes to mount for caching model weights. The kubelet on the assigned node turns that spec into a real running container and keeps it alive according to the restartPolicy. For the LLM app from Chapter 1, the pod is the unit that owns the inference process and its writable cache directory (see Code Walkthrough).

The pod lifecycle, end to end

When you run kubectl apply, your pod does not jump straight to Running. It transitions through Pending (scheduler looking for a node), then ContainerCreating (kubelet pulling the image and setting up the network namespace), then Running. If the container exits cleanly it lands in Succeeded; if it exits non-zero it goes Failed; if it keeps crashing the kubelet wraps it in CrashLoopBackOff with exponential backoff. Knowing where a pod is stuck tells you why: stuck Pending is a resource or scheduling problem (especially common for GPU pods), stuck ContainerCreating with ImagePullBackOff is a registry/tag problem, repeated CrashLoopBackOff is an application problem inside your LLM code.

Verifying the pod from outside the cluster

Reaching Running is necessary but not sufficient — your LLM pod also has to answer HTTP requests on its inference port. In a development cluster without an external load balancer, kubectl port-forward opens a TCP tunnel from localhost to the pod's container port, letting you curl /generate directly. This is the cheapest possible end-to-end check that the image from Chapter 1 actually runs and serves traffic on the cluster (see Code Walkthrough).

Code Walkthrough

The snippet below demonstrates all three concepts in a single flow: it builds the pod manifest for the Chapter 1 LLM image, applies it to the cluster, and watches the lifecycle transitions until the pod is Running. The walkthrough deliberately uses the Python kubernetes client rather than raw YAML so every field is explicit and the watch loop can surface the lifecycle in real time.

Code snippetpython
1from kubernetes import client, config, watch
2
3NAMESPACE = "llm-dev"
4POD_NAME = "llm-server"
5
6def build_llm_pod():
7    container = client.V1Container(
8        name="llm-inference",
9        image="gcr.io/genai-course/llm-server:ch1",
10        image_pull_policy="IfNotPresent",
11        ports=[client.V1ContainerPort(container_port=8080, name="http")],
12        env=[
13            client.V1EnvVar(name="MODEL_NAME", value="gemma-2b"),
14            client.V1EnvVar(name="DEVICE", value="cuda"),
15            client.V1EnvVar(name="HF_HOME", value="/model-cache/huggingface"),
16        ],
17        resources=client.V1ResourceRequirements(
18            requests={"cpu": "2", "memory": "8Gi", "nvidia.com/gpu": "1"},
19            limits={"cpu": "4", "memory": "16Gi", "nvidia.com/gpu": "1"},
20        ),
21        volume_mounts=[
22            client.V1VolumeMount(name="model-cache", mount_path="/model-cache")
23        ],
24    )
25    return client.V1Pod(
26        api_version="v1",
27        kind="Pod",
28        metadata=client.V1ObjectMeta(
29            name=POD_NAME,
30            namespace=NAMESPACE,
31            labels={"app": "llm-server", "model": "gemma-2b"},
32        ),
33        spec=client.V1PodSpec(
34            containers=[container],
35            volumes=[client.V1Volume(
36                name="model-cache",
37                empty_dir=client.V1EmptyDirVolumeSource(size_limit="10Gi"),
38            )],
39            restart_policy="Always",
40        ),
41    )
42
43def watch_until_running(timeout=180):
44    v1 = client.CoreV1Api()
45    w = watch.Watch()
46    for event in w.stream(
47        v1.list_namespaced_pod,
48        namespace=NAMESPACE,
49        field_selector=f"metadata.name={POD_NAME}",
50        timeout_seconds=timeout,
51    ):
52        pod = event["object"]
53        phase = pod.status.phase
54        reasons = []
55        for cs in pod.status.container_statuses or []:
56            if cs.state.waiting:
57                reasons.append(f"WAITING:{cs.state.waiting.reason}")
58            elif cs.state.running:
59                reasons.append("RUNNING")
60            elif cs.state.terminated:
61                reasons.append(f"TERMINATED:{cs.state.terminated.exit_code}")
62        print(f"[{event['type']}] phase={phase} {' '.join(reasons)}")
63        if phase in ("Running", "Failed"):
64            w.stop()
65            return phase
66    return "Timeout"
67
68if __name__ == "__main__":
69    config.load_kube_config()
70    client.CoreV1Api().create_namespaced_pod(NAMESPACE, build_llm_pod())
71    final = watch_until_running()
72    print(f"Final phase: {final}")

Once the watch reports Running, port-forward to the pod and hit the inference endpoint:

Code snippetbash
1kubectl port-forward pod/llm-server 8080:8080 -n llm-dev &
2curl -X POST http://localhost:8080/generate \
3  -H "Content-Type: application/json" \
4  -d '{"prompt": "Explain Kubernetes pods in one sentence", "max_tokens": 64}'

You'll know it works when the watch loop prints phase=Running with a RUNNING container reason, and the curl call returns a JSON completion from the Chapter 1 model. If you instead see WAITING:ImagePullBackOff, fix the image tag; if you see WAITING:CrashLoopBackOff, run kubectl logs llm-server -n llm-dev to read the failure from inside the container.

Do's and Don'ts

Do's

1. ✓Do pin the image tag from Chapter 1 explicitly — using :latest makes it impossible to tell whether a CrashLoopBackOff is your code or a silently updated image.
2. ✓Do set GPU requests equal to limits — Kubernetes does not overcommit GPUs, and asymmetric values are silently rejected by some node configurations.
3. ✓Do watch the pod with kubectl get pod -w or the Python watch loop — polling with repeated kubectl get hides the order in which states transitioned, which is exactly the information you need to debug.

Don'ts

1. ✗Don't treat Running as healthy — a process can be running and still failing every inference request. Always verify with a real /generate call.
2. ✗Don't use kubectl port-forward for anything beyond debugging — it bypasses load balancing and dies with your terminal; promote to a Service in the next chapter.
3. ✗Don't omit the model-cache volume_mount — without a writable cache path, every restart re-downloads weights and the pod will appear to hang in ContainerCreating for minutes.