Design evaluation dataset schema with Pydantic — lab audio overview

Host: Welcome back. You're in GenAI Evaluation, Safety and Governance — a course about making sure the AI systems your team ships actually behave the way you expect, and keep behaving that way over time. This chapter is about evaluation dataset curation: how you build the test sets that tell you whether your language model is doing its job. And today's skill — designing the shape of those test sets — is something every team hits the moment they try to measure quality seriously. Expert: Here's the scenario. Picture a mid-size company — maybe two hundred engineers — that has rolled out a customer support assistant powered by a hosted language model. Things are going fine, until one day the product team swaps in a newer, cheaper model. Suddenly, refund questions start getting weird answers. Nobody notices for three weeks. Why? Because the team had no standard set of test questions with expected answers that they ran before every change. They had a pile of example questions in a spreadsheet, another pile in somebody's notebook, a third pile in a shared document — and no two people agreed on what a test case even looked like. One person wrote just a question. Another added an expected answer. A third added a difficulty rating. When they tried to automate evaluation, half the entries broke the script. This is the mess that evaluation dataset curation solves. Before you can measure anything, you need a shared, strict definition of what a test case is — what fields it must have, what values are allowed, and how to catch bad entries the moment they sneak in. Host: Makes sense. So this is exercise one of three in this chapter — the starting point. The chapter overview laid out the big picture of evaluation data. Now we get specific. What exactly are we building? Expert: You're building two things that work together. First, a blueprint — a strict definition of what a single evaluation test case looks like. Think of it like a form with required fields: the input question, the expected answer, what category of task it is, how difficult it is, and so on. Second, a checker — a piece of code that reads a file full of test cases, one per line, and verifies that every single one of them fills out the form correctly. If anything is missing or malformed, the checker tells you exactly which row has the problem and what's wrong with it. The library you'll use to define the blueprint is called Pydantic — it's a popular Python tool for describing the shape of data and automatically enforcing that shape. You give it a description of what you expect, and it does the checking for you. The file format you'll read from is called JSONL — that stands for JSON Lines, which just means a plain text file where each line is one self-contained record. It's the standard format for evaluation and training data because you can stream through it one record at a time without loading the whole thing into memory. Host: And the key idea — the conceptual aha — what should click before someone writes a single line of code? Expert: The aha is this: your schema is a contract. It's not documentation that sits in a wiki and goes stale. It's executable. The moment someone tries to add a test case that doesn't fit, the code refuses it and tells them why. That changes the whole team dynamic. Instead of a data scientist reviewing every pull request to an evaluation file by hand, the rules enforce themselves. And there's a second layer to this: some fields can't just be any string. Task category, for example, should only be one of a small, known set — maybe classification, summarization, extraction, question answering, and generation. Difficulty should be one of easy, medium, or hard. You'll encode those allowed values as what's called an enumeration, or enum for short — a fixed list of permitted options. That way nobody can sneak in a typo like "summerization" and quietly poison your results six months later. Host: Before we wrap — what's the tricky part? What trips people up on this one? Expert: Two things. First, people tend to make the blueprint too loose at the start — they mark too many fields as optional because it feels flexible and friendly. Don't. Start strict. It's much easier to loosen a rule later than to tighten one after a thousand test cases have been written against a permissive version. Second, when your checker finds a bad row, the error message has to be useful. If the checker just says "validation failed" and stops, the person who ran it has no idea which row out of ten thousand is the problem. So build the checker to keep going after it finds an error — collect all the problems, tag each one with its row number and the specific field that failed, and report them together at the end. That single design decision is the difference between a tool people love and a tool people curse. Host: Perfect. So after this exercise, what can the listener actually do? Expert: After this, you'll be able to take a messy pile of evaluation examples from across your organization and impose order on it — a single, enforced definition of what counts as a valid test case, with meaningful error reporting when something's off. That's the foundation every downstream evaluation tool in your stack will rest on. Your team can use it as the backbone for everything from regression testing to model comparison dashboards. And in the next exercise, you'll build on this directly: you'll write a generator that produces test cases automatically, spread across those task categories and difficulty levels you just defined, so you're not stuck hand-writing every example. The schema you build today is what that generator will produce against tomorrow. Good luck, and thanks for listening.