Containerizing LLM Applications — chapter audio overview

Podcast Script: Containerizing LLM Applications Host: Welcome back. You're listening to Chapter 1 of 12 in Kubernetes Essentials for GenAI Engineers, and this first chapter is titled "Containerizing LLM Applications." Now, before we get into the details, let me set the stage. Your team has invested in building real production AI capability — the kind of depth where you don't just call an API from a notebook, you ship a service that runs reliably in a cluster alongside everything else your organization depends on. This chapter is the foundation for that. Picture this scenario: you've written a small Python service that calls a large language model. It works beautifully on your laptop. You hand it to a teammate, or you try to run it on a staging server, and suddenly nothing works. A missing library, a different Python version, an environment variable that was set in your shell but nowhere else. That's the problem this chapter solves — and it's the reason every production LLM system in the world ships as a container. You'll practice this in six hands-on exercises, but first, let's build the mental model. We'll walk through writing the application, packaging it with Docker, configuring it safely, orchestrating it with supporting services, and publishing it to a registry. Let's start with the application itself. Expert: Happy to. So let's begin with what we're actually building. The application is a small web service written in Python. It has one main job: accept a text prompt from a user, forward that prompt to Google's Gemini model — which is the large language model we're using in this course — and return the generated text back to the caller. Two tools do the heavy lifting here. The first is FastAPI, which is a Python library for building web services. Think of it as the part that listens for incoming HTTP requests and sends responses back. It's popular because it validates incoming data automatically and is very fast. The second tool is Pydantic, a Python library for defining data shapes and validating them. When a request comes in, Pydantic checks that the prompt field is present, that it's not empty, and that it doesn't exceed a reasonable length. If any of that fails, the request is rejected before any call to Gemini happens — which matters because every Gemini call costs money and counts against your rate limits. Now, inside this application we have three layers. The first layer handles HTTP — that's FastAPI's job. The second layer is what I'll call the Gemini client, which owns the actual conversation with the Gemini API. It sends the prompt, waits for the response, handles timeouts, and translates any errors into something the caller can understand. The third layer, and this is the one that matters most for containerization, is configuration. Every value that might change between environments — the Gemini API key, the model name, the endpoint URL, the request timeout — lives in a settings module that reads from environment variables at startup. Here's why that matters. An environment variable is just a named value that the operating system hands to your application when it starts. Your shell has them, your container has them, and Kubernetes has them. By reading configuration from environment variables, the exact same application code runs in development against a local proxy, in testing against a mock endpoint, and in production against the real Gemini API — no code changes, no rebuilds. Pydantic has a feature specifically designed for this called BaseSettings, which reads the environment, validates the types, checks required fields are present, and enforces ranges. For example, the timeout must be between 5 and 120 seconds; the temperature between zero and two. If the Gemini API key is missing, the application refuses to start and tells you clearly what's wrong. That fail-fast behavior is critical because in a Kubernetes cluster, a service that starts but can't actually work is worse than one that crashes immediately — it wastes resources and confuses health checks. The last piece is a health endpoint. This is a simple URL the application exposes that just returns "healthy" without calling Gemini. Health checks run every ten to thirty seconds, and if every check made a real API call, you'd be burning thousands of billable requests per day doing nothing but confirming the service is alive. So the health endpoint deliberately avoids the upstream model. Host: Okay, so we've got a Python service with validated inputs, externalized configuration, and a cheap health check. The key takeaway: every design decision here was made with containerization in mind. Now — how do we actually package that service so it runs the same way everywhere? That's what a Dockerfile is for. Let's go there. Expert: Right. So Docker — let me introduce it properly — is a tool that packages an application along with everything it needs to run, into a single self-contained unit called a container image. That image can then run on any machine that has a container runtime installed, and it behaves identically every time. A Dockerfile is just a text file with a list of instructions that tells Docker how to build that image, step by step. Things like: start from this base Linux image, copy these files in, install these Python packages, run this command when the container starts. Now, the critical concept here is layers. Every instruction in the Dockerfile creates a layer — an immutable snapshot of the filesystem at that point. Docker caches each layer independently. If you change an instruction, that layer and every layer after it gets rebuilt, but everything before it is reused from cache. This is why the order of instructions matters enormously. The rule of thumb: put things that change rarely at the top, and things that change often at the bottom. So the base image comes first. Then the system-level user creation. Then you copy in the file that lists your Python dependencies — called requirements.txt — and install them. Only after that do you copy in your actual application code. Why? Because your application code changes every commit, but your dependency list changes maybe once a week. With this ordering, Docker reuses the cached dependency layer on almost every build, and a code-only change rebuilds in about two seconds instead of sixty. The second big idea is the multi-stage build. The Gemini Python library depends on some packages with compiled C extensions — things that need a full compiler toolchain to install. If you build everything in one stage, all those compilers end up in your final image, bloating it to around one and a half gigabytes. A multi-stage build splits this into two phases. The first stage — the builder — uses a full Python base image with all the compilation tools, installs the dependencies into a virtual environment, and then is discarded. The second stage — the runtime — uses a slim base image, just the Python interpreter and essential libraries, and copies only the finished virtual environment over from the builder. The final image drops from one and a half gigabytes to under three hundred megabytes. That's a five-fold reduction, which matters directly when Kubernetes schedules a new container and has to pull the image across the network. Before we leave the Dockerfile, one more critical file: the dockerignore file. When you run the build command, Docker packages up your current directory and sends it to the Docker engine. Without a dockerignore file, that package includes your dotenv file with the API key in it, your git history, your local virtual environment built for the wrong operating system, and all your test files. Any of these can cause a problem — from secret exposure, to image bloat, to broken imports inside the Linux container because you copied in a macOS compiled library. The dockerignore file works just like a gitignore file: you list patterns to exclude, and Docker never sees those files at all. One last production detail: run your container as a non-root user. By default, the process inside a container runs as the root user, which means if there's ever a container escape vulnerability, the attacker has root access to the host machine. In your Dockerfile, you create a dedicated application user and switch to it before the application starts. Kubernetes production clusters enforce this — images that require root will be rejected at admission time. Host: So to recap: Dockerfile instructions become cached layers, multi-stage builds give you a slim runtime image, the dockerignore file keeps secrets and junk out of the build, and you always run as a non-root user. Now we've got a blueprint. But a blueprint isn't a running service. What happens when we actually build it, tag it, and run it — and how do we pass in things like the API key safely? Expert: Great question, because this is where a lot of teams make expensive mistakes. Building the image is straightforward — you run the Docker build command, point it at your current directory, and give it a tag. A tag is a human-readable name like "gemini-app" version "1.0.0" that you use to refer to the image later. When you run the build, Docker walks through the Dockerfile, checks the cache at every step, and produces an image sitting in your local image store. Now running the image. This is where configuration comes into play. The container needs three things from the outside world. First, port mapping: the application listens on port 8000 inside the container, but that port is isolated from your machine unless you explicitly publish it. So you tell Docker to forward port 8000 on your laptop to port 8000 inside the container. Second, environment variables: this is how the Gemini API key, the endpoint URL, the model name, and the timeout get injected. You pass them on the command line, and they appear inside the container's process environment, where Pydantic's BaseSettings reads them at startup. Third, resource limits: you can cap the container's memory and CPU usage, which is important because a burst of concurrent LLM requests can balloon memory, and you don't want one misbehaving service starving everything else on the host. Now the crucial rule for API keys. Never, ever bake an API key into the image itself. Never put it in a build argument — those values are recoverable from the image metadata by anyone with pull access. Never put it as a default environment value in the Dockerfile — same problem, it gets baked into the layer. Always inject the API key at runtime, from your shell environment or from a secure secret store. Inside the container, the key exists only in the running process's memory, which is the only acceptable place for it. Let's also talk about tagging strategy, because this is where rollback discipline is born. An image tag is just a label — it's mutable. Anyone with push access can reassign a tag to point to a different image. The only truly immutable identifier is the image digest, which is a cryptographic hash of the image contents. For practical operations, though, you use three kinds of tags. First, a semantic version like "1.2.0" — three numbers meaning major, minor, and patch. Major increments for breaking changes, minor for new features, patch for bug fixes. You never reassign a semantic version tag, which means rollback is deterministic: if version 1.2.0 ships a broken prompt template, you change your deployment to reference 1.1.0 and you know exactly what runs. Second, a Git commit SHA — that's the short hash identifying the exact source code that built the image. This gives you traceability when debugging: if an LLM output looks wrong in production, you can map the running container back to the exact commit, check the prompt code, and reason about what changed. Third, a tag called "latest" for development convenience only. Never, ever deploy production workloads using the "latest" tag — it's mutable, and Kubernetes caches images by tag, which means different nodes in your cluster can end up running different code under the same label. That's a nightmare to debug. And one last thing: always verify a pushed image by pulling it back on a clean environment and running the health check. An image that builds and runs on your Apple Silicon laptop might fail on production Kubernetes nodes that use a different processor architecture. Catch that before you deploy, not during an incident. Host: This is really landing. We've got immutable images, runtime-injected secrets, and disciplined tagging. Now — a single container is a nice demo, but real production LLM systems never run alone. They sit next to proxies, caches, and other services. How do we express that whole stack as one cohesive thing we can stand up and tear down? Expert: That's where Docker Compose comes in. Compose is a tool that reads a single text file — usually called compose dot yaml — and uses it to start an entire multi-service application on a single machine. You declare each service, its configuration, how they network together, and the dependencies between them. Then one command brings the whole stack up, and another tears it down cleanly. For our Gemini application, the typical stack has three services. The first is the application itself — the FastAPI service we already built. The second is an API proxy. A proxy is a service that sits between your application and an external API and handles cross-cutting concerns: rate limiting, key rotation, retries, caching. We're using a lightweight open-source proxy called Nginx here, which is a very common web server and reverse proxy. The third service is Redis, which is an in-memory data store commonly used as a cache. In this case, Redis stores Gemini responses so that repeated prompts don't have to make another paid API call. Compose handles three critical things that you'd otherwise have to wire up manually. The first is networking. All three services live on a shared internal network, and within that network, each service is reachable by its service name as a hostname. So your application talks to the proxy by the name "api-proxy" and to the cache by the name "redis" — no hardcoded IP addresses, no localhost references. This is identical to how Kubernetes service discovery works, which makes the transition in later chapters almost seamless. The second thing Compose handles is configuration layering. You point each service at an environment file containing the shared settings, and then you can override specific values per service inline. The third, and honestly most important, is startup ordering based on health. You can say: don't start the application until the proxy and the cache both pass their health checks. Without this, all three services start at the same time, and your application tries to connect to the proxy before it's ready, leading to confusing connection-refused errors on the first few requests. Each service also gets its own health check defined right in the Compose file. For the application, it's the health endpoint we already built. For Redis, Compose runs a ping command against it. For the proxy, Compose makes a small HTTP request to a health path. Compose polls these at regular intervals, and the dependency ordering respects their status. The development workflow with Compose is beautiful. You run one command — Compose up — and your whole stack is running. Edit a Python file, run Compose up again targeting just the application service, and it rebuilds only that one container, leaving the proxy and cache untouched with their state preserved. Cached Gemini responses from your last test session are still there, saving you both time and money. When you're done, Compose down stops everything cleanly. And here's the architectural payoff. The Compose file is essentially a local rehearsal for your Kubernetes deployment. Each service becomes a Deployment object in Kubernetes. The internal network becomes a cluster network. The environment file becomes a combination of ConfigMaps — for non-sensitive values — and Secrets, for the API key. The health checks become what Kubernetes calls readiness probes. The named volume that persists Redis data becomes a persistent volume claim. By building the Compose stack correctly now, the migration to Kubernetes in the next chapter becomes a translation exercise, not a redesign. Host: Beautiful — so Compose turns a collection of containers into a coordinated application, and the patterns you learn here carry straight over to Kubernetes. The final step in the chapter: how do we get these images out of your laptop and into a place where a production cluster can actually pull them? And along the way, let's hear the production wisdom and the top do-not-ever mistakes. Expert: Absolutely. A container image that only exists on your laptop is useless to a cluster. You need to push it to a container registry, which is a remote storage service that hosts versioned images with access controls. The one we use in this course is Google Artifact Registry, which is Google Cloud's managed registry service, but Docker Hub, AWS's Elastic Container Registry, and Azure Container Registry all work identically. You authenticate once using your cloud credentials, then the Docker push command uploads your image layer by layer. Because of the way Docker stores layers, if the registry already has a layer from a previous push — say, the base Python image — it skips uploading that one and just records the reference. So pushing a code-only change transfers maybe fifty kilobytes, not a hundred and twenty megabytes. The recommended push workflow applies three tags to the same image digest: the semantic version, the Git commit SHA, and optionally "latest" for local development. Then you verify by pulling the image back on a clean environment, starting a container, and confirming the health endpoint responds. This catches architecture mismatches, missing certificate bundles, and other runtime failures before production does. Now, the production wisdom. If you remember nothing else from this chapter, remember three things. First: externalize all configuration through environment variables, and validate them at startup using Pydantic. A missing API key should crash the container immediately with a clear error, not silently fail on the first user request. Second: use multi-stage builds and a careful dockerignore file. Image size directly impacts how fast new containers reach readiness during scaling events, and an unfiltered build context is how API keys leak into image history. Third: deploy production workloads using pinned semantic version tags, never "latest." The "latest" tag is mutable, Kubernetes caches by tag, and you will eventually ship a cluster where different nodes run different code under the same name. And the top do-not-ever list. Do not put API keys in build arguments or default environment values in the Dockerfile — they persist in image metadata and anyone with pull access can extract them. Do not copy your entire project directory without a dockerignore file — you will eventually leak a dotenv file or a local virtual environment that breaks in the container. Do not set overly aggressive request timeouts — Gemini responses can take ten to fifteen seconds for complex prompts, and a five-second timeout works in development and fails in production. Do not skip running as a non-root user — Kubernetes production clusters will reject root containers at admission time. And do not skip the post-push verification — an image that builds locally can still fail after a push-pull cycle due to architecture mismatches or missing certificate bundles that only manifest when you make a real HTTPS call to the Gemini API. One more production concern worth naming: secret management. Environment variables are visible through container inspection and through the Kubernetes API server. For production API keys, you graduate to Kubernetes Secrets mounted as files, and you update your settings code to read from a file path when one is provided, falling back to the environment variable for local development. That's a small code change that pays off enormously in security posture. Host: Incredible depth there. Now — in the exercises, you'll practice every one of these layers hands-on. Each exercise has its own audio overview that goes deeper into the specifics. The first exercise is writing the Python application that calls Gemini and returns structured responses. The second is writing the Dockerfile and building the image. The third runs the container with environment-based configuration. The fourth uses Docker Compose to stand up the application alongside supporting services. The fifth tags images with semantic versions and pushes them to a registry. And the sixth teaches you to debug containers using the inspection, logs, and exec commands — essential troubleshooting skills you'll use daily. Host: Let's close out. You now understand how to containerize a Python LLM application from source code to registry — producing portable, reproducible artifacts. You now understand why multi-stage builds, externalized configuration, and disciplined tagging aren't just nice-to-haves but the difference between a system you can operate at scale and one that surprises you at the worst possible moment. And you now understand how the Docker-native stack you build here — services, networks, environment files, health checks — maps directly to Kubernetes concepts coming in the next chapter. This is the depth your team needs when the conversation turns to how your organization actually ships LLM services to production. The chapter quiz will focus on the GenBodha-branded Gemini application, the Python and FastAPI patterns you learned, what the WORKDIR instruction does inside a Dockerfile, and which tagging strategies are safe for production. Pay special attention to the difference between build-time and runtime configuration, and to why "latest" is unsafe in a cluster. In Chapter 2, we take the tagged image you pushed to the registry and run it as your first pod inside a real Kubernetes cluster — turning today's container into tomorrow's scalable service. See you there.