Compare Docling output quality across document types and complexity levels — lab audio overview

Host: Welcome back to the GenBodha lab podcast. Today we're diving into the third and final lab of objective one-one in the Document Ingestion with VLMs chapter, part of our GenAI Data Pipelines course. The lab is titled "Compare Docling output quality across document types and complexity levels." Expert, this one feels like the capstone of the trio. What makes it distinct? Expert: Great framing. Labs one and two were about getting Docling to actually parse things. Lab one was pure PDF on CPU with layout analysis. Lab two expanded to DOCX, PPTX, and HTML through the unified pipeline. But both of those stopped at "did it parse?" Lab three asks a harder question: "did it parse well?" The student is building a quality evaluation framework — essentially a measurement harness that scores Docling's structural fidelity across all four formats and across simple versus complex document samples. Host: So we're moving from extraction to evaluation. What exactly will the student construct? Expert: The core deliverable is a comparison framework with three scoring dimensions. First, heading preservation — does the extracted document tree match the expected hierarchy of H1, H2, H3 levels? Second, table extraction accuracy — are rows, columns, and cell boundaries intact, or did the parser flatten a table into a wall of text? Third, content completeness — what percentage of the source text actually made it through. The student writes a DocumentQualityScorer class that takes a Docling DocumentConverter result and an expected-structure manifest, then emits per-dimension scores plus an aggregate fidelity number. Host: What patterns or classes from Docling should they focus on? Expert: They'll be working heavily with the DoclingDocument object model — iterating over its items, checking the label field to distinguish headings from tables from paragraphs, and walking the hierarchy. They'll use the TableItem structure to pull out row and column counts and compare against ground truth. There's also a FormatComparator pattern worth calling out: a single interface that runs the same input through the four format paths and normalizes outputs for apples-to-apples scoring. The key design pattern here is separation of parsing from evaluation — Docling does extraction, their scorer does judgment. Host: Let's talk real-world relevance. Who actually needs this in production? Expert: Picture a RAG platform team at a mid-sized enterprise SaaS company — say, a legal tech startup ingesting ten thousand contracts per day, or an insurance company processing claims documents. Their pipeline pulls PDFs, Word docs, PowerPoint decks, and scraped HTML from customer portals. A data engineer or ML engineer is responsible for making sure the chunks flowing into the vector database are actually usable for retrieval. Host: And what happens without a quality evaluation framework like this? Expert: This is where it gets painful. Without evaluation, you ship Docling into production, it works beautifully on your test PDFs, and then three weeks later your retrieval accuracy quietly drops ten points. Why? Because your customer started uploading PowerPoint decks where slide titles aren't tagged as headings, so your chunking strategy flattens everything into one giant blob. Or a complex PDF with nested tables gets parsed as free text, and now your LLM is hallucinating numbers. The blast radius is the entire downstream RAG quality — and the ML engineer gets paged when the product team notices answers have gone sideways. By then you've also poisoned your vector store with thousands of bad embeddings that need to be re-ingested. Host: Who owns this workflow day to day? Expert: Typically a data engineer or ML platform engineer on an ingestion team. Their weekly routine includes running regression suites when Docling releases a new version, when the company onboards a new customer with a new document style, or when someone proposes switching from Docling to an alternative parser. The framework from this lab becomes the regression harness. Without it, you're flying blind on every model or parser upgrade. Host: What trade-offs will the student wrestle with in this lab? Expert: The big one is scoring methodology. Do you use exact-match on heading text, or fuzzy matching with something like Levenshtein distance? Exact-match is brittle — whitespace differences tank your score. Fuzzy is more forgiving but can mask real extraction bugs. The student has to pick a threshold and defend it. A related trade-off is how to weight the three dimensions. Is a missing table worse than a missing heading? For a legal contract, yes. For a marketing slide deck, maybe not. The lab pushes students to make these weightings explicit and configurable rather than hardcoded. Host: Any other design decisions worth highlighting? Expert: Two more. First, how to handle the "complex document" category. The lab provides both simple and complex samples, and the student has to decide whether to report scores separately or roll them up. Separate reporting reveals that Docling might be ninety-five percent accurate on simple PDFs but only seventy percent on multi-column academic papers — and that's a critical operational signal. Second, whether to include timing and memory metrics alongside fidelity. Quality without performance context is incomplete. A parser that's perfect but takes thirty seconds per page is useless at ten thousand documents a day. Host: How does this build on labs one and two specifically? Expert: Lab one gave them the CPU-based PDF path and DocumentConverter fluency. Lab two taught them the unified pipeline across DOCX, PPTX, and HTML. Lab three reuses both — the student instantiates converters from lab one and two, but now wraps them in a measurement layer. If they skipped the earlier labs, they'd have no intuition for which formats are likely to fail on which dimensions. Lab three is where that hands-on knowledge becomes a defensible engineering artifact they can hand to their team. Host: What should students focus on to get the most from this lab? Expert: Don't treat the scoring code as a throwaway exercise. Write it like you'd write a production evaluator — clean interfaces, configurable weights, JSON-serializable outputs, and clear per-dimension breakdowns. The code you write here is a template you'll carry into your first real ingestion pipeline job. Also, spend time inspecting why Docling fails on complex documents. Those failure modes are the real lesson. Host: Excellent breakdown. Students, head into lab three ready to measure, not just extract. Good luck.