Document Ingestion with VLMs — chapter audio overview

Podcast Script: Document Ingestion with VLMs Host: Welcome back. You're listening to Chapter 1 of 18 in GenAI Data Pipelines — Document Ingestion with Vision Language Models. Before we dive in, let me set the scene. Picture this: your team has been asked to build a system that answers questions over ten years of company contracts, research PDFs, and scanned invoices. Everyone's excited about the retrieval model, the embeddings, the clever prompts. But nothing in that system matters until you can reliably get clean, structured text out of documents that were never designed for machines to read. That's the problem this chapter solves — and it's a core competency for any team building production AI. Your organization invested in this training because they need engineers who don't just call APIs, but who build the extraction infrastructure the rest of the pipeline stands on. Over the next six hands-on exercises, you'll build a complete document ingestion pipeline using three complementary tools. But first, let's build the mental model. We'll walk through each extraction backend, the unified document model that ties them together, the routing logic that picks the right backend per document, and the storage layer that makes all of this auditable in production. Let's start with the foundation: local document parsing. Expert: Great framing. So let's begin with the first of our three extraction backends — Docling. Docling is an open-source Python library from IBM Research. Think of it as a universal document reader that takes any PDF, HTML file, Word document, PowerPoint, or even an image, and produces one consistent structured result — a tree of typed content blocks like headings, paragraphs, tables, and figures, with the reading order preserved. Why does that matter? Because in a typical enterprise, the same logical document might exist in three different formats. Without a unified parser, you'd maintain three separate extraction pipelines, each with its own bugs and edge cases. Docling collapses that into a single entry point. Now, how does it actually work under the hood? When you hand Docling a PDF, it runs a pipeline of stages. First, it parses the raw bytes to recover any embedded text layer. Then it performs layout analysis — figuring out where the headings, paragraphs, tables, and figures sit on each page. For scanned pages without a text layer, it invokes an OCR engine — OCR stands for Optical Character Recognition, which is the classic technique for turning pixels into characters. Docling lets you plug in EasyOCR or Tesseract here. Then there's a specialized model called TableFormer, a transformer trained specifically to recognize table structure — which cells belong to which rows and columns, including merged cells. Finally, everything is serialized into that hierarchical document tree. The critical configuration choice is when to run OCR. Many real-world PDFs are hybrid — the first hundred pages are born-digital with clean embedded text, and the last ten are scanned appendices. If you force full-page OCR on every page, you waste compute and sometimes replace good text with worse text. The recommended configuration runs OCR only on pages that lack an extractable text layer, which gives you accuracy on scanned pages without the performance penalty on digital ones. For batch processing, Docling streams results as they complete. You hand it a list of file paths, it returns results one by one, each tagged with a status — full success, partial success where some pages worked, or failure. That lets your pipeline keep moving even when individual documents fail. And one production gotcha worth calling out now: Docling normalizes different input formats into the same output schema, but normalization is not lossless. HTML documents carry CSS hints about structure that are encoded differently than Word heading styles or PDF font-size heuristics. So even after Docling gives you a unified representation, you often need a small post-processing step to reconcile heading levels across source formats — for example, shifting everything down so the document always starts at heading level one. Host: So Docling gives us a local, no-cost, deterministic parser for born-digital documents and clean scans. But we know real-world document collections include handwritten annotations, complex multi-column layouts, and pages where rule-based parsers just fall apart. That's where the second backend comes in. Expert: Right — vision language models, or VLMs. Let me unpack that term first. A vision language model is a multimodal neural network, meaning it accepts both images and text as input. GPT-4o from OpenAI and Gemini from Google are the two best-known examples. When you show a VLM a page image, it doesn't go through the traditional OCR stages of deskewing, character recognition, and layout analysis as separate steps. Instead, it interprets the whole page as a visual scene in a single forward pass — it sees the heading hierarchy, the table cells, the figure captions, and the reading order simultaneously, because it learned those spatial relationships from millions of document images during pretraining. So why would you ever use something slower and more expensive than Docling? Because VLMs excel exactly where rule-based parsers fail. Handwritten annotations on a scanned form. A research paper with figures interleaved between two text columns. An invoice where the "total" label sits three inches away from the total value. A VLM reads these like a human does, using visual context. The trade-off is cost and latency. VLM extraction runs ten to fifty times more expensive per page than Docling, and a single high-resolution page image consumes one to four thousand tokens of context depending on the model and detail setting. How do you actually use one? You render each page as an image, encode it, and send it to the API along with a prompt that specifies the output structure you want. This is really a prompt engineering problem — the same page will produce wildly different results depending on whether you ask for "all the text" versus "a JSON object with headings, paragraphs, tables, and figure captions." Production prompts are specific. They describe the schema the model should return, they set the temperature to zero to maximize determinism, and they use the response format feature that forces the model to return valid JSON matching that schema. Now, a critical production concern: VLMs can hallucinate. Because they're generating text rather than recognizing pixels, they can invent content that looks plausible but isn't on the page. So you cannot treat VLM output as ground truth the way you can with traditional OCR. You need quality checks — comparing extracted content against visual features of the page, flagging low-confidence regions, and ideally benchmarking your VLM pipeline against a ground-truth dataset before trusting it in production. The usual benchmarking metric is Character Error Rate — the edit distance between extracted text and ground-truth text as a fraction of the total length. Production pipelines typically aim for error rates below two percent on born-digital documents and below five percent on scanned ones. When you benchmark Docling against a VLM on clean born-digital PDFs, you often find the VLM is marginally more accurate but costs a hundred times more. That's exactly the evidence you need to justify routing most documents through the cheaper backend and reserving VLMs for the hard cases. Host: So Docling is your default for clean documents, VLMs are your specialist for complex layouts. But there's a middle ground — what if you need enterprise-grade OCR at scale without running any of this on your own infrastructure? Expert: That's the role of our third backend — Google Document AI. This is a fully managed cloud service from Google that provides pre-trained processors for document understanding. When I say "processor," I mean a versioned extraction model you create inside your Google Cloud project — you pick the type, you get a resource name, and you send documents to it. There are several processor types. There's a plain OCR processor, a Form Parser for key-value extraction, and specialized ones for invoices and receipts. The most useful for general ingestion pipelines is the Layout Parser processor. It produces a hierarchical document structure — paragraphs grouped into sections, tables with properly attributed cells, reading order inferred across multi-column pages — which is considerably richer than basic OCR output. Why would you choose Document AI over self-hosting Docling? Three reasons: scale, compliance, and operational simplicity. Document AI handles automatic scaling — you never provision servers or tune memory. It provides processor versioning, which means you can pin your pipeline to a specific model version while evaluating newer versions in parallel. And it's compliant with the regulatory frameworks enterprise customers care about. The cost model is per page, and there are two invocation modes. Synchronous processing sends one document and waits for the result — fine for low volume or interactive use cases. But for anything beyond a few hundred documents a day, you must use batch processing. Synchronous calls hit rate limits fast, and they carry request overhead on every call. Batch processing reads documents from Google Cloud Storage, processes them in parallel inside Google's infrastructure, and writes results back to Cloud Storage as JSON. A thousand-document batch typically finishes in fifteen to thirty minutes. One subtle thing about the Document AI response format: all extracted text is stored in a single string on the document object, and every structural element — every paragraph, every table cell — references back into that string with start and end index offsets. So when you build your parsing utilities, you need a small helper that dereferences those anchors to pull out actual content. It's a clever design for storage efficiency, but it requires careful index management when you parse. Host: Okay — three backends, three output formats. That's where the architecture starts to get interesting, because downstream systems cannot deal with three different representations. So how do we unify them, and how do we decide which backend to use for any given document? Expert: Two pieces of the puzzle, and they work together. Let's start with the unified document model. The core problem: Docling produces its tree of typed nodes, Document AI produces those text-anchor-referenced layouts, and a VLM returns whatever JSON shape your prompt asked for. Downstream consumers — your chunking logic, your embedding pipeline, your citation generator — cannot care which backend produced the data. So we define a single canonical schema using Pydantic, which is a Python library for data validation and structured models. The schema captures a document as a list of pages, each page holds a list of content blocks, each block has a type — paragraph, heading, table, figure, list item — and critically, every block carries provenance metadata. Provenance means the audit trail: which extractor produced this block, what version of that extractor, what page it came from, what bounding box coordinates on that page, and what confidence score the extractor assigned. This provenance is not optional. In regulated industries — financial services, healthcare, legal — you must be able to prove that any piece of extracted text came from a specific region of a specific page. Without provenance, you cannot satisfy audit requirements and you cannot debug retrieval regressions. You also embed a schema version field in every document payload, because your schema will evolve over the years and you need old documents to remain loadable without a forced migration. Then, for each extraction backend, you write an adapter — think of it as a translator. One adapter translates Docling's output into the unified model, another translates Document AI's response, another translates the VLM JSON. Adding a fourth backend later just means writing a fourth adapter. No downstream code changes. Now the routing layer. Not every document needs the same backend. You build a lightweight classifier that inspects each incoming document and produces a complexity score. It looks at things like: does the document have an extractable text layer, or is it scanned? How many pages? Are there embedded images? Are there table-like block structures? A fast tool called PyMuPDF gives you all of that in under a hundred milliseconds per document without full parsing. Then the router applies thresholds. Clean born-digital PDFs route to Docling. Large batches, or documents with moderate complexity, route to Document AI. Complex scanned documents with high complexity scores route to a VLM — but only if the page count is manageable, because VLM cost scales linearly with pages. And finally, you layer in fallback chains. If Docling fails or produces low-quality output, the pipeline automatically retries with Document AI. If Document AI fails, it tries a VLM. The document never silently disappears; it gets a second chance through a different backend before the pipeline raises an error. Host: So we have extraction, normalization, and intelligent routing. The last architectural layer is actually persisting these extracted documents in a way that's durable, queryable, and auditable. That's the storage layer. Expert: Exactly — and this is what separates a prototype from a production system. We use two complementary stores. Google Cloud Storage, usually abbreviated GCS, is Google's object storage service — durable, cheap, version-aware. We write the full extracted document content there as JSON blobs. PostgreSQL, which is an open-source relational database, serves as the metadata catalog — it holds the structured fields we need to query across the whole corpus. Why both? Because JSON blobs in object storage are great for durability but terrible for answering questions like "show me every document extracted by Docling version two with confidence below 0.8 in the last week." That's a relational query, and it belongs in PostgreSQL. Here's how they fit together. Each document gets written to GCS at a deterministic path — something like documents, then source type, then date, then a hash of the source URI, then a version number. That path schema gives you prefix-based lifecycle policies — you can retain invoices for seven years and web scrapes for ninety days by configuring rules on those prefixes. It gives you time-based partitioning. And the hash-based document ID makes re-extraction idempotent — running the same source through the pipeline twice produces the same path, so the second write is a no-op. On the PostgreSQL side, you design three tables. A documents table holds document-level metadata. An extraction jobs table records every extraction attempt with its configuration, status, cost, and timing. And a lineage edges table implements a directed graph connecting source artifacts to output artifacts through transformations. That lineage graph is the thing that turns your pipeline from a black box into an observable system — when a retrieval returns wrong information three months from now, you trace back from the embedding, to the chunk, to the extracted document, to the source file, and you find exactly where the error originated. The ingestion transaction must be idempotent and consistent. The pattern is: check PostgreSQL for an existing record at the same schema version. If it exists, skip. Otherwise, write to GCS first, then commit the PostgreSQL metadata with the extraction job and lineage edge in a single transaction. If the GCS write fails, no metadata record is created. If PostgreSQL fails after GCS succeeds, the next retry finds the existing blob and just commits the metadata. That ordering matters. Now let me close with production wisdom — three things to remember if you remember nothing else. First, do not send every document through VLM extraction by default. VLMs cost ten to fifty times more per page than traditional parsing. A hundred-thousand-document corpus with fifteen pages each is 1.5 million page extractions — routing everything through a VLM runs forty-five thousand dollars, while routing eighty percent through Docling and escalating only the hard cases costs nine thousand. Second, always store lineage and provenance metadata on every extracted block. Without it, you cannot debug, re-extract on model upgrades, or satisfy audit requirements. And third, never hardcode extraction configuration. OCR settings, VLM prompts, quality thresholds — all of it belongs in environment variables or a configuration service, so you can tune extraction behavior without redeploying. Host: That's a lot of ground, and the good news is you'll get to build all of it. In the six hands-on exercises for this chapter, you'll first extract documents using Docling's unified parser. Then you'll process documents with VLM-based understanding using hosted APIs from OpenAI and Google. You'll use Google Document AI for managed OCR and layout parsing. You'll design the unified document model that normalizes all these extraction outputs into one schema. You'll build the routing system that selects the optimal extraction method per document. And finally, you'll store extracted documents in GCS with PostgreSQL metadata tracking. Each exercise has its own audio overview that goes deeper into the implementation details. Host: Let's close with three takeaways. You now understand how three extraction backends — Docling, Google Document AI, and vision language models — complement each other across cost, accuracy, and latency. You now understand why a unified document model with provenance and lineage metadata is the architectural centerpiece that lets downstream systems stay decoupled from extraction details. And you now understand how intelligent routing plus durable storage in GCS and PostgreSQL turns an extraction script into a production pipeline. You have the depth to evaluate extraction approaches for your team's document pipeline and explain the trade-offs to stakeholders. The chapter quiz will test your understanding of Docling's pipeline configuration, the role of IBM's library in multi-format parsing, when VLM-based extraction is justified, and how the unified document model supports multiple backends. Pay close attention to the routing thresholds and the cost-accuracy tradeoffs. In the next chapter, Chapter 2 on Data Cleaning and Quality Agents, we'll build on this foundation — once documents are extracted, you still need to detect encoding issues, clean up corrupted text, and validate structural quality before the content reaches your embedding pipeline. See you there.