Build cost-aware routing balancing quality vs API spend per document — lab audio overview

Host: Welcome back to the GenAI Data Pipelines podcast. Today we're diving into a lab that sits right in the middle of our document ingestion routing sequence — Lab 2 of 3, titled "Build cost-aware routing balancing quality vs API spend per document." I've got our Expert here to walk us through what makes this lab distinct. Let's start with that — how does this one differ from Lab 1? Expert: Great question. Lab 1 was about classification — you built a classifier that looks at a document and says "this is a simple text PDF" or "this is a dense scanned form with tables." That's pure signal extraction. Lab 2 takes that signal and does something economically meaningful with it. The student builds an ExtractionRouter that makes a decision: given this document's complexity class, and given how much of my daily budget I've already burned, which extraction method should I actually invoke? Lab 3 will then layer fallback chains on top when that chosen method fails. So Lab 2 is the economic brain in the middle. Host: So this is really where cost meets quality. Expert: Exactly. And this is the skill that separates hobbyist pipelines from production ones. Think about a mid-size insurance carrier processing two hundred thousand claims documents a day. If every single document goes to GPT-4 Vision or Claude with high-resolution image inputs, you're looking at tens of thousands of dollars a day in API spend. But maybe sixty percent of those documents are clean typed PDFs where Tesseract OCR gives you ninety-eight percent accuracy at effectively zero cost. The router is what captures that savings. Host: What does the student actually build in this lab? Expert: Two main components. First, the ExtractionRouter class itself — it takes a document plus a complexity signal from the Lab 1 classifier, and it has a routing policy that picks between extraction methods. Typically you'll have a tiered set — something like a cheap local OCR tier, a mid-tier cloud OCR like AWS Textract, and a premium tier using a vision language model. The router consults a cost table, looks at the remaining daily budget, and picks the highest-quality method the budget allows for that document's complexity. Host: And the second component? Expert: The CostTracker. This is the piece that estimates per-document API spend before the call goes out, records actuals after, and enforces a daily budget ceiling. The pattern here is really important — you want a pre-flight estimate so the router can reason about "can I afford this," and you want post-flight reconciliation because actual token counts vary. The tracker typically exposes methods like estimate_cost, record_spend, and remaining_budget, and it persists state so it survives restarts. Host: What are the key classes and patterns the student should focus on? Expert: The router itself is a strategy pattern — you have an abstract ExtractionMethod interface and concrete implementations for each tier. The router holds a registry of those methods with their cost profiles. The CostTracker is essentially a token bucket with accounting. And I'd call out one subtle pattern: the router should emit a routing decision object — not just invoke the method directly — so that logging, metrics, and the Lab 3 fallback logic can all see why a particular method was chosen. Host: Let's talk about the production scenario. Who's getting paged when this isn't built right? Expert: The ML platform engineer, usually at two in the morning. The classic failure mode is what we call cost runaway — somebody uploads a batch of ten thousand multi-page scanned contracts, every single one gets routed to the premium VLM tier because there's no budget enforcement, and by sunrise you've blown through the monthly API budget in six hours. Finance notices, the CFO emails the CTO, and the on-call engineer has to hard-stop the pipeline. Without a CostTracker enforcing daily caps, there's literally no circuit breaker between your ingestion queue and your vendor invoice. Host: And the inverse failure? Expert: Equally bad — being too aggressive on cost. You route everything to cheap OCR, your extraction accuracy drops from ninety-five to seventy-eight percent, and downstream the claims adjudication model starts making wrong decisions on real customer policies. That's a quality incident, and it lands on the data engineer who owns the pipeline. This is why the "balancing" part of the lab title matters — you're not minimizing cost, you're maximizing quality subject to a cost constraint. Host: Who uses this skill daily? Expert: Data engineers and ML engineers on document intelligence teams — think legal tech, insurance, healthcare claims, or any RAG pipeline ingesting enterprise documents. Platform engineers at AI infrastructure companies build generalized versions of this as a service. And increasingly, SREs are involved because cost is now a reliability concern — a runaway spend incident gets treated with the same severity as a production outage. Host: What's a trade-off the student should wrestle with in this lab? Expert: The big one is: do you reserve budget per document class, or do you run a single global budget? Global is simpler and lets high-value documents consume more when traffic is light, but it means a flood of cheap documents can starve the premium tier later in the day. Per-class reservation is fairer but more complex and can leave budget unspent. There's also a threshold question — when remaining budget drops below, say, twenty percent, do you downgrade all routing, or only the borderline cases? The student will make these calls and see the consequences in the test scenarios. Host: Any final focus areas? Expert: Three things. First, make the cost table externally configurable — pricing changes and you don't want to redeploy code for that. Second, log every routing decision with its rationale — this is what your finance team will ask for. Third, make the CostTracker thread-safe if the pipeline is concurrent, because double-spending a budget under race conditions is a real bug. Nail those three and Lab 3's fallback chains will slot in cleanly on top. Host: Perfect setup for Lab 3. Thanks for walking us through it. Expert: Anytime. Happy routing.