Define a format-agnostic document schema with typed content blocks — lab audio overview

Host: Welcome back to the GenAI Data Pipelines podcast. Today we're diving into a very specific lab from the Document Ingestion with VLMs chapter — lab one of three under the objective "Design a unified document model normalizing all extraction outputs." The lab is titled "Define a format-agnostic document schema with typed content blocks," and the project is called define-formatagnostic-document-schema. Expert, why is this the right place to start? Expert: Because everything downstream depends on this contract. In this objective, lab two builds extraction adapters for Docling, VLMs, and Google Document AI, and lab three validates the unified model end-to-end across all three methods. But none of that works until lab one nails the schema. If the schema is wrong, every adapter leaks its idiosyncrasies upward, and your retrieval layer ends up doing ad-hoc cleanup forever. Host: So what exactly is the student building in this lab? Expert: A Pydantic-based unified document model. At the top, there's a Document class that holds a list of content blocks, plus document-level metadata — source URI, ingestion timestamp, the extraction method used, and an overall confidence score. The interesting work is in the typed content blocks. The student defines a discriminated union — a TextBlock for paragraphs and headings, a TableBlock with structured rows and columns, an ImageBlock with a caption and optional OCR text, and a KeyValueBlock for form-style extractions like invoices and contracts. Each block carries its own ExtractionMetadata: the originating method, a per-block confidence, the page number, and a bounding box in normalized coordinates. Host: And what makes this "format-agnostic"? Expert: The schema never mentions Docling, VLMs, or Document AI in its type names. It only talks about what a block *is*, not where it came from. The provenance lives in the metadata, not the type. So a TableBlock extracted by Docling and a TableBlock extracted by Gemini Vision are the same class — downstream code treats them identically, but can still audit where each came from. Host: Let's make the production scenario concrete. Who actually needs this? Expert: Picture a mid-sized fintech onboarding team or an insurance claims platform ingesting hundreds of thousands of PDFs, scanned forms, and contract images daily. A data engineer or ML platform engineer typically owns this pipeline. Without a unified schema, what happens is this: the Docling path returns markdown with its own heading conventions, the VLM path returns JSON with bounding boxes in pixel coordinates, and Document AI returns its own proto-style entities with different confidence semantics. Your retrieval team writes three branching code paths. Your evals team can't compare methods because the outputs aren't commensurable. Host: And when that breaks in production? Expert: The blast radius is ugly. A downstream change — say, enabling the VLM path for a new document type — silently shifts bounding box conventions from normalized to pixel coordinates. Suddenly highlight rendering in the customer-facing UI is off by a factor of a thousand. Support tickets spike, and the on-call SRE gets paged at two in the morning for "document previews broken." They can't roll back cleanly because three services already consumed the malformed blocks into caches. The post-mortem invariably says: "we needed a strict schema at the ingestion boundary." That's what this lab is. Host: What Pydantic patterns should the student focus on? Expert: Three things. First, discriminated unions using a literal `block_type` field — this is what makes parsing safe and lets Pydantic dispatch to the right class automatically. Second, custom validators for confidence scores, making sure they're always in zero-to-one range regardless of how the source method expressed them. Document AI gives you a float already, but some VLM responses give percentages or categorical labels like "high" — the validator normalizes these at the boundary. Third, `model_config` with `extra="forbid"` so unknown fields from a misconfigured adapter fail loudly instead of silently dropping data. Host: What's a real trade-off the student will face? Expert: The big one is strictness versus flexibility. If you make the schema too strict — say, every TableBlock must have a header row — you'll reject legitimate extractions where the header is ambiguous. Too loose, and you're back to ad-hoc cleanup. The lab pushes the student toward making optional fields *truly* optional with sensible defaults, while making the structural invariants — block type, page number, confidence — non-negotiable. Another trade-off is coordinate systems. Do you store bounding boxes as normalized floats zero-to-one, or as pixel coordinates with a page dimension? Normalized is more portable across methods but loses precision on huge pages. The lab walks through why normalized wins for a format-agnostic schema. Host: How does this set up lab two? Expert: Lab two is where the student writes adapters — one per extraction method — that each return a `Document` instance. The whole point is that those adapters should be thin. If the schema in lab one is designed well, each adapter is mostly field-mapping and coordinate normalization. If the schema is designed poorly, the adapters become giant transformation pipelines with their own bugs. So lab one is really about getting the contract right so lab two becomes almost boring. Host: And lab three? Expert: Lab three runs real documents — a PDF, a scanned image, a form — through all three methods and asserts the outputs conform to the same schema and are semantically comparable. That's only possible because lab one locked down the types. Host: Any last advice for the student? Expert: Spend time on the ExtractionMetadata class. It feels like a side detail, but in production this is what lets your ML engineers debug why a RAG query returned garbage — they trace a bad retrieval back to a specific block, see it came from the VLM path at confidence 0.42, and know to route that document type to Document AI instead. Provenance is not optional at scale. Host: Great framing. Students — head into define-formatagnostic-document-schema, read the TODOs carefully, and remember: the schema you write here is the contract the next two labs depend on. Expert: Design it like a public API, because within this objective, it is one. Host: Thanks for listening.