Chat Completion API with Streaming — chapter audio overview

Podcast Script: Chat Completion API with Streaming Host: Welcome back to the Full-Stack GenAI Applications podcast. This is Chapter 1 of 18, and today we're building something you'll use in every production chat feature you ever ship — token-by-token streaming for large language models. Picture this scenario while you're driving: a user types a question into a chat box, hits send, and then stares at a blank screen for eight seconds. That's the default behavior when you just return a single response from a model. Eight seconds feels broken. Now imagine instead that words start appearing one at a time, almost immediately — that's streaming, and it's the difference between an app that feels alive and one that feels dead. This is a core competency for any team building production AI systems. Your organization has invested in your GenAI training because they need engineers who can build the infrastructure behind AI features, not just call an API once and wait. You'll practice this across five hands-on lab exercises, each targeting a different model provider — but first, let's build the mental model. We'll cover the streaming protocol itself, then four provider integrations, then the production lessons that separate a demo from a real system. Let me bring in our expert. Host: So the foundational idea here is something called Server-Sent Events. Before we talk about any model provider, can you explain what that protocol actually is — and why we use it instead of WebSockets for chat streaming? Expert: Happy to. Server-Sent Events — SSE for short — is a W3C web standard that lets a server push a continuous stream of messages to a client over a single, long-lived HTTP connection. The key idea is that the server keeps the connection open and sends small text frames as events happen, instead of bundling everything into one response. Think of it like a ticker tape — the tape keeps feeding out one update at a time, and the reader processes each update the moment it arrives. Now, the wire format is almost laughably simple. Each event is a line that starts with the word "data" followed by a colon and a payload, then a blank line. That blank line — two newline characters in a row — is what tells the browser "this event is complete, hand it off to the application." Miss that blank line and the browser buffers forever, waiting for a terminator that never comes. The content type on the response is set to "text slash event-stream," and that string is the signal to browsers and proxies that this is a streaming event feed, not a regular JSON payload. People often ask why not use WebSockets. WebSockets are bidirectional, meaning both sides can talk at any time. That's powerful, but it requires a protocol upgrade handshake that corporate proxies often block, and it adds complexity you don't need for chat. A chat response is fundamentally one-way — the model generates tokens, the browser displays them. SSE rides on plain HTTP, needs no handshake, survives corporate networks, and the browser even has a built-in client for it called EventSource that handles reconnection automatically. On the server side, we use FastAPI, which is a modern Python web framework that speaks asynchronous Python natively. FastAPI has a response class called StreamingResponse — think of it as a pipe you hand to FastAPI, and whatever you push into that pipe gets flushed to the client immediately. To feed the pipe, we use something called an async generator. An async generator is a Python function that uses the "yield" keyword to produce values one at a time, but it can also pause and wait on network calls without blocking anything else. So the recipe is: your async generator receives tokens from the model provider, formats each token as an SSE frame, yields it, and FastAPI flushes it straight to the browser. There's one more critical detail — client disconnect detection. When a user closes their tab mid-response, the TCP connection drops, but your generator doesn't automatically know. If you ignore this, your code keeps consuming tokens from the provider — which you pay for — for content nobody will ever read. FastAPI exposes a method on the request object that tells you whether the client is still connected. You check it on each loop iteration, and if the client is gone, you break out and stop the upstream call. That single check is the difference between a hobby project and a system that doesn't bankrupt your team. Host: Okay so we have the protocol and the FastAPI plumbing. The first provider in the chapter is OpenAI. What's special about their streaming contract, and what's the adapter pattern we're building? Expert: OpenAI set the de facto standard here, and most other providers imitate it. When you call their chat completions endpoint with streaming turned on, instead of getting one big response object, you get an async iterator — something you can loop over with "async for" — that yields small objects one at a time. Each of these objects is called a chat completion chunk, and the important field inside it is called the delta. Delta just means "the difference" — the new content since the last chunk. Usually a delta is a single token, sometimes two or three, and occasionally it's empty or missing. The first chunk in a stream typically has no content at all — it just announces the role, usually "assistant." The middle chunks carry the actual text, one little fragment at a time. The final chunk sets a field called finish reason to a value like "stop" or "length," which tells you why the model stopped generating. "Stop" means natural completion. "Length" means the model hit the maximum token budget you configured and got cut off. "Content filter" means OpenAI's moderation layer intervened. Your code needs to distinguish these because a truncated response is not the same as a completed one, and users deserve to know the difference. Now, why do we wrap all of this in an adapter? An adapter is just a translator — a class that knows how to talk to one specific provider and converts their output into a format your rest of the system understands. The streaming adapter for OpenAI takes their raw chunk objects and emits a unified event frame that looks the same no matter which provider generated it. So our frontend code, or any downstream consumer, never has to know whether the tokens came from OpenAI, Gemini, Anthropic, or somewhere else. That's what we mean by provider-normalized frames — one envelope, many sources. Two production details worth calling out. First, you must use the asynchronous client, not the synchronous one. A synchronous call blocks the entire event loop, which means every other user hitting your server stalls until that one response finishes. Since streaming calls can last thirty seconds or more, one synchronous call can freeze your whole service. Always the async client inside an async endpoint. Second, OpenAI has a stream option called "include usage" that you can turn on. When it's enabled, a special final chunk arrives after the text is done, carrying the prompt tokens and completion tokens used. That's the only reliable way to get token counts in streaming mode — without it, you're guessing. You'll want this for cost accounting and for rate limit enforcement. And when you receive that usage chunk, note that its choices list is empty — if your code assumes there's always a choice to read, it will crash. Guard for that empty list. Host: Got it — OpenAI gives you a flat sequence of deltas and you just extract content until you see a finish reason. Now Gemini is a different animal because of something called thinking budget. What does that mean and how does it change the streaming flow? Expert: Great question, because this is where things get interesting. Google's Gemini 2.5 Flash introduced a feature where the model can spend some of its token budget on internal reasoning before producing its visible answer. You, the developer, control how many tokens it's allowed to spend on this internal thinking. That control is called the thinking budget, and you pass it through a configuration object on each request. Here's the mental picture. Imagine you ask a colleague a hard question. If they blurt out the first thing that comes to mind, you get a fast but shallow answer. If they pause, think carefully, and then speak, you get a slower but better answer. Thinking budget lets you dial that behavior on the model. Set the budget to zero, and Gemini skips reasoning entirely and starts emitting visible tokens immediately — roughly comparable latency to the previous non-thinking model. Set the budget to eight thousand, and the model may silently reason for two to five seconds before the first visible token appears. This creates a two-phase streaming response. During the thinking phase, the model emits what are called thought tokens. These represent its internal chain of reasoning. Then it transitions into the text phase and emits the actual user-facing answer. The Gemini client library exposes this distinction — each piece of content that arrives has a flag telling you whether it's a thought or a final output token. Your adapter has to decide what to do with thought tokens. You might suppress them entirely for a clean user experience. You might route them to a separate event type so a debugging interface can display them. Or you might forward them as-is if your product explicitly shows reasoning. There's a subtle thing that catches people off guard. When thinking is enabled, Gemini forces the model's temperature setting to one point zero and ignores whatever you pass in. If you set temperature to point three expecting deterministic output, it silently gets overridden. Your adapter should handle this by only applying custom temperature when thinking is disabled. Practical guidance on choosing a budget. For simple queries — greetings, lookups, short factual questions — set the budget to zero. You get low latency and lower cost. For complex reasoning tasks — debugging code, multi-step math, planning — bump it up to a few thousand tokens. The cost matters because thinking tokens count toward your output token billing. A request with an eight-thousand token budget that fully uses it costs roughly two to three times more than the same request with thinking disabled. So ideally, your application routes requests dynamically — simple stuff bypasses thinking, hard stuff gets it. And one more thing unique to Gemini — the streaming response does not send an explicit finish reason field like OpenAI does. You detect completion by the iterator naturally running out. That means your "done" sentinel frame — the final marker you send to the browser signaling the stream is over — has to be emitted unconditionally after the loop ends, not triggered by a specific flag inside the stream. Host: That's a really useful framing — thinking budget is a cost-latency-quality knob you turn per request. Next up is Anthropic's Claude, and I understand their streaming format is structurally different. What should listeners expect? Expert: Yes, Anthropic takes a different design approach, and it's worth understanding why. OpenAI and Together both give you a flat stream of chunks with a single delta field. Anthropic gives you a typed event stream, where each event has a specific type and its own payload shape. You can think of it like the difference between a single-track conveyor belt versus a structured envelope with labeled sections. Let me walk through the event lifecycle. A Claude stream starts with a "message start" event, which announces the beginning of the response and includes metadata like the message identifier and how many input tokens you sent. Then you get a "content block start" event, which declares a new content block is opening — a content block is a container for one logical piece of output, typically text. Inside that block, you receive a series of "content block delta" events, each carrying a small piece of text. When that block is done, you get a "content block stop." Finally, there's a "message delta" event with the stop reason and output token usage, followed by a "message stop" that ends the whole thing. Why the extra structure? Claude can produce multiple content blocks in a single response. One block might be text, another might be a tool call, a third might be extended thinking content. The block-level framing lets the client distinguish these cleanly. For basic chat streaming, you only care about content block delta events where the inner type is text, and you ignore the rest of the ceremony. But the structure is there when you need it. There are a few normalization gotchas when you map Claude onto your unified frame format. Anthropic's stop reasons use different words than OpenAI. Where OpenAI says "stop," Anthropic says "end turn." Where OpenAI says "length" for hitting max tokens, Anthropic says "max tokens." Your adapter should translate these onto a common vocabulary so your frontend doesn't have to know who generated the response. Another difference — the Anthropic SDK exposes its stream as an async context manager. That means you open it with an "async with" block, and when the block exits, the connection closes automatically, even if your consumer breaks out early. This is actually really nice for resource safety, especially paired with client disconnect detection. If the user closes their browser and your generator breaks out of its loop, the context manager unwinds and the connection to Anthropic closes cleanly. No leaked sockets, no lingering inference on the provider side. One last thing on retries. Anthropic's client has built-in retry logic for rate-limit errors and overloaded responses, but those retries only happen before the first token arrives. Once streaming has begun, a mid-stream failure cannot be retried — retrying would cause duplicate tokens to appear in the output. So the rule is: retry at connection time, and if the stream breaks mid-flight, emit a structured error frame and let the user choose whether to start over. That retry-before-first-token pattern is a principle you'll apply across all providers. Host: Interesting — Claude's typed events force a bit more ceremony but give you cleaner structure. Now the fourth provider is a little different because it's not really a new API — it's Together.ai hosting an open-weight model. Walk us through that. Expert: Right, and this is where the adapter pattern really pays off. Together.ai is a managed inference platform. They host open-weight models — meaning models whose weights are publicly available, like Meta's Llama family — on their own GPU infrastructure, and they expose them behind an API that mirrors OpenAI's contract exactly. Same endpoint shape, same request format, same response chunks. This is called an OpenAI-compatible API, and it's become a de facto standard across the industry. The model we're using here is Llama 4 Maverick. It's a mixture-of-experts model. Quick explanation — a mixture-of-experts, or MoE, model is one where the total parameter count is huge, but each individual token only activates a small subset of those parameters. Llama 4 Maverick has around 400 billion total parameters but only about 17 billion active on any given token. This gives you the capacity of a massive model with the throughput of a much smaller one. Running it yourself would require serious GPU infrastructure, which is exactly why teams reach for a hosted service. The elegant part is that you use the same OpenAI Python client you used for GPT. You just change two things when you construct it. First, you override the base URL to point at Together's API gateway. Second, you pass in a Together API key instead of an OpenAI key. The rest of your streaming code — iterating chunks, pulling deltas, watching for the finish reason — is identical. That's the power of the compatibility layer. A single adapter class with a configurable base URL lets you swap providers without rewriting any streaming logic. A few gotchas specific to Together. First, they don't always return usage counts in streaming responses, so if you need token accounting you'll need to count tokens on the client side or make a separate non-streaming call. Second, when their servers are overloaded, they sometimes return an error as a JSON object embedded inside the stream, rather than as an HTTP status code. Your adapter needs to recognize that shape and convert it to an error frame — otherwise the error message leaks into the chat as if it were generated text. Third, time-to-first-token tends to be a bit higher than OpenAI — often four hundred to eight hundred milliseconds rather than one fifty to three hundred — because the model runs on distributed GPU nodes. So set your read timeouts generously, around one hundred eighty seconds, to avoid false failures on long generations. The big architectural lesson — once you have a clean provider abstraction, adding a new provider is mostly a config entry. Your routing layer reads a provider name from the request, looks up the adapter, and dispatches. Adding a fifth or sixth provider becomes a single-file change. Host: So four providers, one unified streaming contract. Before we wrap up, let me ask the most important question for someone shipping this to production. What are the two or three things that will absolutely bite us if we get them wrong? Expert: I'll give you the short list. If you remember nothing else from this chapter, remember these. First — buffering intermediaries will silently destroy your streaming experience. Reverse proxies like Nginx, cloud load balancers, and CDNs all buffer HTTP responses by default. They wait for a chunk of response to fill a buffer before forwarding it, which means your carefully streamed tokens get batched and delivered in clumps. The fix is setting specific headers on every streaming response — a cache-control header set to "no-cache," and a header called "X-Accel-Buffering" set to "no." That second one specifically tells Nginx to disable buffering for this response. Forgetting these headers is the single most common reason developers say "streaming works locally but all the tokens arrive at once in production." It's a silent failure — no error message, just bad user experience. Second — always send a done sentinel frame. The OpenAI convention is to send a final frame with the literal text "data colon space bracket DONE bracket" before closing the stream. Your browser needs a reliable way to tell "the response is complete" from "the connection dropped unexpectedly." Without the sentinel, clients either hang waiting for more data or misinterpret a network timeout as an incomplete response. Yield the sentinel after the last content frame, every time, on every provider. Third — handle client disconnects explicitly. Wrap your yield loop in a try-finally pattern so that when the user closes their tab, your generator cleans up — closes the upstream HTTP connection, cancels any watchdog tasks, releases resources. Without this, a user navigating away from a long response leaves your server happily generating and paying for tokens that will never be read. At scale this is real money. A few quick don'ts. Don't use the synchronous client inside an async endpoint — it blocks the entire event loop. Don't forget to set the media type on your streaming response to "text slash event-stream" — the wrong content type is another silent failure where the data arrives but the client can't parse it as events. Don't expose raw provider error messages to users — map errors to a controlled set of codes and log the full details server-side. And don't hardcode provider base URLs across your codebase — put them in a configuration object so a single URL change doesn't trigger a multi-file hunt. One more that people skip — send periodic keep-alive pings during long thinking phases. Cloud load balancers drop idle connections after about sixty seconds. If Gemini is quietly reasoning for ninety seconds without emitting a visible token, the load balancer kills the connection. A tiny SSE comment frame every fifteen seconds keeps the connection alive and the user's session intact. Host: That's incredibly practical. Now let's talk about the hands-on part. What will listeners be building in the labs? Expert: Five exercises, each one focused. The first builds the core FastAPI streaming endpoint itself — you'll wire up the streaming response, write the async generator, and format SSE frames correctly. The second plugs in OpenAI's GPT-4o and handles the chat completion chunk deltas. The third brings in Gemini 2.5 Flash with thinking budget control, so you get hands-on experience with the reasoning versus latency trade-off. The fourth integrates Anthropic's Claude and teaches you to consume its typed event stream. The fifth adds Llama 4 Maverick through Together.ai, showing the power of the compatibility layer. Each lab has its own audio overview that goes deeper on the code-level details. Host: Perfect. So let's close out. You now understand three big things. You understand how the Server-Sent Events protocol works and why it's the right choice over WebSockets for unidirectional token streaming. You understand how to build a provider-normalized adapter layer that hides the quirks of OpenAI, Gemini, Anthropic, and Together.ai behind one unified event format. And you understand the production concerns — client disconnect detection, proxy buffering, keep-alive pings, and retry semantics — that separate a working demo from a real system. You now have the depth to design the streaming layer for your team's chat features and to explain the trade-offs to stakeholders — especially around Gemini's thinking budget, which is a direct cost lever. The chapter quiz will focus on which protocol choice fits streaming chat, why SSE is HTTP-based rather than requiring a new transport like WebSockets, and the mechanics of FastAPI's StreamingResponse and the browser's EventSource API. Pay particular attention to the buffering and content type gotchas. In the next chapter, we move from direct provider integrations to a unified gateway using LiteLLM, which will let you route across providers without writing a new adapter every time. The mental model you built today is exactly what that gateway abstracts over. See you in Chapter 2.