Data Pipelines - Configurable Generator Pipeline with Gemini — lab audio overview

Host: Welcome back. You're in LLM Foundations — a course about the core Python patterns that power large language model applications. Large language models, or LLMs, are the AI systems behind tools like ChatGPT. This chapter is about generators and iterators — two Python techniques for processing data one piece at a time instead of loading everything at once. And that matters because when you're working with AI, the data rarely fits in memory all at once. Expert: Let me paint the picture. Imagine you're on a small data team at a mid-sized company — maybe fifteen engineers. Your product team says, "We want to analyze a hundred thousand customer support tickets using AI. Summarize each one, flag the urgent ones, and save the results." So you write a straightforward script. It calls the AI, gets a summary back, moves to the next ticket. You run it. It loads all hundred thousand tickets into memory. Then it tries to hold all hundred thousand AI responses in memory too. Your laptop freezes. Or worse — it runs on a server, eats through your cloud budget, and crashes two hours in with nothing saved. This is the moment every data engineer meets generators. Generators let you process data as a flowing stream — one item in, one item out, next item in, next item out — so memory stays flat no matter how much data you push through. In this exercise, you'll build something called a configurable pipeline. Think of it as an assembly line where each station does one job, and items flow through one at a time. The AI model you'll connect to is Google's Gemini — Google's family of large language models, accessed over the internet through a simple interface. Host: Good. So this is exercise one — your starting point for the chapter. The overview introduced the big ideas; now we're getting our hands on them. So what exactly will we build? Expert: You'll build a data pipeline you can configure — meaning you can plug different stages together like LEGO bricks. The pipeline has four jobs. First, source the data — in this case, ask Gemini to generate a stream of items, maybe product descriptions, maybe customer responses, whatever the scenario calls for. Second, filter — keep only the items that match some rule you define, like "only items longer than fifty words" or "only items that mention pricing." Third, transform — take each surviving item and change it somehow, maybe shortening it, maybe extracting a specific field. Fourth, collect — gather the final results into a list you can use. Here's the key idea, the conceptual aha: each stage doesn't do its work all at once. Each stage is lazy. Lazy means it does nothing until the next stage asks for the next item. So when the collector asks for one result, that request travels backward up the chain. The transformer asks the filter for one item. The filter asks the source for items until one passes the rule. The source asks Gemini for one more piece of data. One item flows through the entire pipeline, end to end, then the next item starts. That's the magic. You never hold the full dataset anywhere. Host: That lazy evaluation idea is the heart of it. Before I let you go start coding — what's the one thing that trips people up here? Expert: The trickiest part is realizing that if you never ask for results at the end, nothing happens anywhere. Beginners wire up the whole pipeline, run their program, and see absolutely nothing — no errors, no output, no Gemini calls. They think it's broken. It's not. The pipeline is sitting there patiently waiting for someone to pull on it. Until the final collecting step actively requests items, every earlier stage stays frozen. The tip: always test by collecting results at the end, even just the first few, and watch them print. If you see output, your pipe is flowing. If you see silence, you forgot to pull. Host: That's a great mental checkpoint. So let's close with what this unlocks. Expert: After this exercise, you'll be able to take any data source — an AI model, a file, a database — and run it through a chain of filters and transformations without ever loading the full dataset into memory. You'll also be able to swap stages in and out without rewriting the whole thing. This is a foundational building block your team can reuse anywhere you're processing AI outputs at scale — batch summarization, content moderation, enrichment jobs, evaluation runs. It's the pattern that separates a prototype that works on ten examples from a system that works on ten million. This is the only exercise in this chapter, so once you finish, you've got the complete pattern — a working piece of code you can bring straight back into your team's architecture discussions when someone asks, "How do we handle this volume?" Thanks for listening, and good luck.