The LLM Client — chapter audio overview

Podcast Script: The LLM Client Host: Welcome back to the show. This is Chapter 11 of 81 in GenAI Agent Engineering, and today we're building the single most important piece of plumbing in any AI agent — the LLM client. In the last chapter we walked through project structure, how to lay out your codebase so it can grow. Now the question is: how does your Python code actually talk to a large language model? Because here's the thing — your organization has invested in your growth as a GenAI engineer, and this is where that investment starts paying off. Anyone can paste an API key into a script. Engineers who understand the client layer — the authentication, the retries, the fallback logic, the token tracking — those are the engineers who build the infrastructure that production systems depend on. You'll practice this in six hands-on exercises spanning OpenAI, Anthropic, Google Gemini, and a unified wrapper that ties them together. But first, let's build the mental model. We'll start with how these APIs actually work, walk through each of the three providers, build a unified interface, handle errors gracefully, and finish with async patterns for high-performance production work. Let's get into it. Host: So before we touch any provider-specific code, help the listener understand what's actually happening on the wire. When my Python application sends a prompt to OpenAI, what's really going on? Expert: Great place to start. Every major LLM provider — OpenAI, Anthropic, and Google — operates on what's called a client-server model. Your application is the client. You send an HTTP request over the internet to the provider's servers, and the server runs the actual language model and sends back a response. You don't host the model yourself. You rent access to it. Now, every request has four ingredients. First, authentication — an API key, which is essentially a secret password tied to your account. The provider uses that key to verify who you are, to track your usage, and to bill you. Second, model selection — you specify which model you want, because each provider offers several. OpenAI has fast economical models and more capable ones. Anthropic has Claude in different tiers. Google has Gemini variants. Third, the messages — the conversation history and the current prompt, structured as a sequence of turns with roles like system, user, and assistant. The system role sets the behavior and personality. The user role is what the human asked. The assistant role is what the model said before. Fourth, the parameters — things like temperature, which controls randomness, and max tokens, which caps the length of the response. Now, the server sends back three things: the generated text, the token usage breakdown for cost tracking, and metadata like a unique request ID. Here's the critical choice every engineer faces: do you talk to these APIs using raw HTTP requests, or do you use the official SDK — which stands for software development kit, basically a Python library the provider ships that handles the HTTP details for you? The answer is almost always the SDK. With raw HTTP, a typo in a field name only shows up at runtime as a mysterious error. With the SDK, your IDE catches it immediately. The SDK also gives you typed error classes — so instead of parsing status codes yourself, you catch a specific exception like a rate-limit error. The SDK handles streaming, connection pooling, and automatic retry with exponential backoff. You get all of that for free. Unless you have a very unusual requirement, use the official SDK. Always. Host: That framing — four ingredients in, three things out — makes it concrete. Now you mentioned three providers. Walk me through what's actually different between them, because I'd expect them to all look basically the same. Expert: You'd think so, and at a high level they are. But the details diverge in ways that bite you if you're not careful. Let's start with OpenAI. You install their Python package with pip, create a client object with your API key, and call the method that creates a chat completion. You pass the model name, the messages array, and your parameters. The response comes back, and you reach into it to get the generated text, the input token count, the output token count, and metadata. Their parameter for maximum response length is called max_tokens. Straightforward. Anthropic — which makes Claude — looks similar on the surface but has three important differences. First, the system prompt is not part of the messages array. It's a separate top-level parameter. So if you're writing code that targets both providers, you have to pull the system message out and pass it differently for Anthropic. Second, the max tokens parameter is required — you cannot leave it out. Third, the response content isn't a single string. It's a list of content blocks, because Anthropic supports multi-part responses that may include different types of content. You have to iterate through those blocks and pull out the text. And their token counts use different field names — input tokens and output tokens, where OpenAI calls them prompt tokens and completion tokens. Same concept, different words. Then there's Google Gemini, which is the most different of the three. Gemini doesn't use a messages array — it uses what it calls contents, which is a list where each turn has a role and a parts array wrapping the text. Its generation parameters live under a separate configuration object, and they use camelCase names — so maxOutputTokens instead of max_tokens. Gemini also has a dedicated chat interface that maintains conversation history automatically, so you don't have to resend the full history on every turn like you do with OpenAI and Anthropic. And the async story is different — OpenAI and Anthropic give you a separate async client class, while Gemini uses a single client that exposes async operations through a property. These differences aren't academic. When a real request fails in production and you're debugging at two in the morning, knowing that Anthropic wants the system prompt separately is the difference between a five-minute fix and a two-hour rabbit hole. Host: Okay, so three providers, three slightly different shapes. That sounds like a recipe for messy application code. How do we prevent every part of our codebase from having to know about these differences? Expert: This is exactly the right question, and the answer is one of the most important patterns in this whole course — a unified interface. The idea is you define one standard shape for a chat message, one standard shape for a response, and one standard contract that any LLM client in your system must honor. In Python, there's a feature called a Protocol, which is a way to define that contract without forcing inheritance. It's structural — any class that has the right methods automatically satisfies it. So you define your protocol, say, a client must provide a generate method that takes messages and returns a unified response. Then for each provider, you write what's called an adapter. An adapter is a thin wrapper that takes your standardized request, translates it into whatever weird shape that specific provider wants, calls the underlying SDK, and translates the response back into your standard shape. The OpenAI adapter takes your unified messages and sends them as-is. The Anthropic adapter pulls out the system message and passes it separately. The Gemini adapter rebuilds the contents list with parts and maps the assistant role to Gemini's model role. Your application code never sees any of this. It just calls generate and gets back a response with content, input tokens, output tokens, model name, and a finish reason. Then on top of that, you typically build what's called a factory — a small object that looks at a string like "openai" or "anthropic" and gives you back the right configured client. So to switch providers, your application changes literally one string. Think of it like electrical outlets. Different countries have different plug shapes, but if you have the right adapter, you plug your laptop into any wall anywhere in the world. The unified interface is your universal adapter. The payoff is enormous. You can A/B test providers. You can fail over from one to another. You can let different teams in your organization pick the model that fits their use case. And none of that business logic has to change. This is the kind of design decision that makes the difference between a proof of concept and a system that can actually evolve with your company's needs. Host: That's a really clean separation. But even with a unified interface, the network is still the network — things fail. What happens when a provider is down or throttling us, and how do we make the system resilient without building something so complicated it becomes its own source of bugs? Expert: Resilience starts with understanding the failure modes. Each provider defines its own set of errors — rate limit errors when you're sending too many requests, timeout errors when the response takes too long, authentication errors when your API key is invalid, bad request errors for malformed parameters, connection errors when the network drops, and internal server errors when the provider itself is having a bad day. OpenAI names them one way, Anthropic names them another, Gemini uses yet a third convention. So the first step is classification — you map every provider's error into your own unified error type. Rate limit, timeout, authentication, invalid request, content filter, connection, server error. Now you can reason about them consistently. Then you split these errors into two categories. Some errors are recoverable — rate limits, timeouts, connection failures, transient server errors. These are worth retrying. Some errors are not — authentication failures, bad requests, content filter violations. Retrying those is pointless and wastes money. So your client recognizes the difference. For recoverable errors, you apply exponential backoff with jitter. That means you wait a short time and retry, and if it fails again, you wait twice as long, and so on. The jitter adds a little randomness so that if many clients all fail at the same moment, they don't all retry in lockstep and hammer the provider. Now, for the next level of resilience, you implement a fallback chain. Instead of retrying the same provider forever, after a few attempts you fall back to a different provider entirely. OpenAI times out? Try Anthropic. Anthropic rate limits you? Try Gemini. This is where the unified interface pays off enormously — because all three providers look the same from your application's perspective, the fallback is a simple loop. Finally, there's the circuit breaker pattern. This is borrowed from electrical engineering. If a provider is failing repeatedly, the circuit trips open, and for a set recovery period, you don't even try that provider — you go straight to the fallback. After the timeout, you let a few test requests through. If they succeed, the circuit closes and normal traffic resumes. The benefit is you stop hammering a provider that's already struggling, which is both more polite and actually helps them recover faster. One warning: never silently swallow errors. Log them with context so you can see patterns. Also, never retry on authentication failures — if your key is bad, retrying a thousand times won't fix it. Host: That's the defense in depth I was looking for — retry, fall back, circuit break. Now let's talk about speed. Modern agents often need to make many calls at once. How do we scale that without blowing up? Expert: This is where async patterns come in, and they're essential for production systems. Normal Python code is synchronous — you call a function, you wait for it to return, then you do the next thing. That's fine when you're making one API call. But imagine you need to summarize a hundred documents. If each call takes two seconds and you run them sequentially, that's over three minutes of wall-clock time where your program is mostly just waiting on the network. Async changes that. Python's async model lets you start many network requests, and while each one is waiting for a response, the event loop — which is the scheduler managing all these in-flight operations — works on the others. So a hundred two-second requests might all finish in a few seconds total. All three providers have async clients. OpenAI and Anthropic ship a dedicated async client class. Gemini exposes async through a property on its regular client. In every case, the API looks almost identical to the synchronous version, except you use the await keyword when you call the method and you use async iteration for streams. There's a standard Python function called gather that takes a list of these async operations and runs them concurrently, giving you all the results when they're done. Now, unbounded concurrency is dangerous. If you fire off ten thousand requests at once, you'll hit the provider's rate limit, and half will fail. The control mechanism for this is called a semaphore, which is a fancy word for a counter that limits how many things can happen at the same time. You configure it for, say, ten concurrent operations. Every request has to grab a slot before it runs, and release the slot when it's done. For longer-term throttling, you layer on a token bucket, which refills a pool of allowed requests every minute. Between the two, you get both instantaneous concurrency control and sustained throughput limits. One more async pattern — streaming. Instead of waiting for the full response to finish generating, you receive chunks of text as soon as they're produced. This dramatically improves perceived latency for users, because they see text appearing immediately rather than staring at a loading spinner for ten seconds. Every provider supports streaming. The shape differs — OpenAI yields chunks with delta content, Anthropic uses a context manager with a text stream, Gemini gives you an iterable of response chunks — but your adapter normalizes all of them into a single async stream your application consumes. This is a feature that feels cosmetic but genuinely changes how users experience your agent. Host: Okay — architecture, providers, unified interface, error handling, async patterns. That's the full tour. Before we head to the labs, what are the three or four production lessons you want every engineer on the team to carry away? What's the "if you remember nothing else" list? Expert: Four things. First — API keys are secrets. Treat them like passwords. Never put them in source code, never commit them to version control, never log them, never embed them in error messages. Store them in environment variables, or better, use a typed settings object that automatically masks the value when it's printed. If you hardcode a key even "just for testing," it ends up in git history, it gets scraped by a bot, and someone in a different time zone is running your credit card up all weekend. I have seen this happen. It is painful. Second — track tokens from day one. Every single API call returns input and output token counts. Capture them. Every provider charges per million tokens, and the rates vary dramatically by model. A single careless loop that sends huge prompts can turn into a five-figure bill overnight. Build a simple cost tracker that records usage per model, multiplies by the pricing table, and alerts you when you cross fifty percent, eighty percent, and one hundred percent of your daily budget. This is not optional for production. And this, by the way, is the entire topic of the next chapter — token economics. Third — log request metadata but not request content. You want to know which provider handled the request, which model, how many tokens, how long it took, whether it succeeded. You do not want raw user messages and raw model outputs in your logs, because those logs leak, and if there's sensitive content in there, you have a compliance problem. Use a hash of the content for correlation — you can trace a request without exposing what was said. Fourth — validate inputs before sending. A surprisingly large category of attacks on LLM systems is prompt injection, where a user embeds instructions like "ignore previous instructions" in their message to hijack the model. A simple validator that checks message length and scans for known injection patterns won't catch everything, but it raises the floor significantly. One final don't — never reuse the client object incorrectly, but also never create a fresh client for every request. Create it once, reuse it. That gives you connection pooling, which is a big performance win. Host: Excellent. So let's connect that to the labs. You're going to work through six hands-on exercises, and each has its own audio overview that goes deeper. Exercise one — you'll initialize and configure an OpenAI client. Exercise two — the same for Anthropic. Exercise three — Google Gemini. Exercise four is the big one where you build the unified client interface we talked about, with adapters for all three providers. Exercise five introduces async patterns for concurrent requests. And exercise six is where you implement provider-specific error handling and the fallback chain. Each exercise builds on the one before, so by the end you'll have a production-quality client library you could drop straight into a real project. Host: Let's wrap up. After this chapter, you now understand three things. First, how LLM APIs actually work under the hood — the client-server model, authentication, messages, parameters, and why the official SDKs are always the right choice over raw HTTP. Second, how to work with OpenAI, Anthropic, and Google Gemini individually, and how to hide their differences behind a single unified interface using the protocol and adapter patterns. Third, how to make your client resilient and fast — with error classification, fallback chains, circuit breakers, async concurrency, semaphores for rate limiting, and secure handling of API keys. You now have the depth to evaluate LLM integration choices for your team's architecture discussions — not just which provider, but how to build the client layer so it can adapt as the landscape shifts. The chapter quiz will test your understanding of the major providers — OpenAI, Anthropic, Gemini — their Python SDKs, the key differences between them, and when to choose which pattern. Pay special attention to the structural differences, especially the system prompt handling and token usage field names. In the next chapter, we move into token economics — how tokens are counted, priced, and optimized — which plugs directly into the client you just learned to build. See you there.