Create DatasetBuilder with coverage analysis — lab audio overview

Host: Welcome back. You're in GenAI Evaluation, Safety and Governance — a course about making sure the large language models your team deploys actually behave the way you need them to in production. This chapter is about curating evaluation datasets. Large language models, by the way, are the AI systems behind tools like chatbots and coding assistants. And this specific skill — knowing whether your test set actually covers the ground you claim it covers — is something every team wrestles with the moment they stop trusting vibes and start trusting numbers. Expert: Let me paint the scenario. Imagine you're on a small platform team, maybe six engineers, at a mid-sized financial services company. Your team just shipped an internal assistant that helps analysts summarize reports, answer questions, and pull out key figures. Leadership asks a fair question: how do you know it's any good? So you put together two hundred test examples, run them through, and report ninety-one percent accuracy. Great number. Then someone in the room asks — how many of those examples were hard? How many were about summarization versus question answering? Were any about numerical reasoning? And suddenly you realize you don't know. Your ninety-one percent might be ninety-one percent on easy summaries and zero percent on the hard numerical stuff — which is exactly what the analysts actually use it for. Without coverage analysis, your evaluation score is a comforting lie. Every serious AI team hits this wall, and the teams that ship production systems learn to measure the shape of their test data, not just the score. Host: So this is exercise three of three in this objective. In the previous exercise you built a generator that produces test examples across five task categories and three difficulty levels — easy, medium, and hard. Now you're extending that work. So what exactly will we build in this final piece? Expert: You're going to build a dataset builder — a piece of code that takes a file full of evaluation examples and turns it into something you can actually trust and measure. The file format is called JSON Lines, which just means one example per line, each line a small structured record. Your builder does three things. First, it loads that file and checks every single entry against a strict definition of what a valid example looks like — the right fields, the right types, nothing missing. The tool doing this checking is called Pydantic, which is a Python library that validates data against a schema and rejects anything malformed. Second, it computes coverage statistics — how many examples fall into each category, how the difficulty levels are distributed, and the average length of the inputs measured in tokens, which are the small chunks of text that language models actually read. Third, it produces a coverage report — a formatted summary, laid out as readable tables, that also flags balance problems. For example, if ninety percent of your examples are easy, the report calls that out as an imbalance your team should fix before trusting any evaluation numbers. Host: Before someone starts coding, what's the tricky part here? The thing that trips people up? Expert: The trap is treating coverage as a single number. People want one score — "our dataset is eighty-two percent covered" — and that number is meaningless. Coverage is always multi-dimensional. You have coverage by category, coverage by difficulty, coverage by input length, and these dimensions interact. You might have perfect category balance but all your hard examples concentrated in one category, which means your hard-difficulty results are really just results for that one task type. The concrete tip: when you compute statistics, compute them per dimension and also look at the cross-sections — the combinations. And when you flag balance issues, don't just check that each category has some examples. Check that each category has examples at each difficulty level. That's the check that catches the hidden gaps. Host: So after this, what can you actually do, and what comes next? Expert: After this exercise, you'll be able to take a raw file of evaluation examples, validate it end to end, and produce a report that tells your team — honestly — what their test set covers and where the holes are. That's a capability you can bring straight back into your team's evaluation pipeline. It's the foundation for every downstream conversation about model quality, because once you trust the shape of your test set, you can trust the scores that come out of it. And since this is the final exercise in the objective, you now have the complete picture — you designed the schema, generated stratified test cases, and built the curation and coverage layer on top. That's a working evaluation dataset system you can bring into your team's architecture discussions as the starting point for production-grade LLM evaluation. Thanks for listening, and good luck with the build.