Evaluation Dataset Curation — chapter audio overview

Podcast Script: Evaluation Dataset Curation Host: Welcome back to the GenAI Evaluation, Safety and Governance course. This is Chapter 1 of 25, and the topic is Evaluation Dataset Curation. If your team has invested in building real GenAI capability — the kind that ships models into production rather than just prototyping against an API — then this chapter is where the engineering discipline actually begins. Because here's the uncomfortable truth: every production language model system has a silent dependency that most teams underestimate. It's not the model. It's not the prompt. It's the evaluation dataset that decides whether a new model version reaches your users. When that dataset is skewed toward easy examples, contaminated with training data, or missing entire task categories, your team approves regressions you cannot detect until customer complaints surface. Picture this scenario: a hosted summarization service evaluated against five hundred hand-curated examples. Four hundred and eighty are single-paragraph news articles. Twenty are multi-document legal briefs. The legal use case drives sixty percent of revenue — and your evaluation score tells you almost nothing about it. Today we fix that. You'll practice this in six hands-on exercises, but first let's build the mental model. We'll walk through stratified sampling, content-addressable versioning, contamination detection, staleness monitoring, and privacy-safe synthetic data. Let's start with the core problem. Expert: Great framing. Let me begin with the foundational idea, which is stratified sampling. The word "stratified" just means "split into layers," where each layer is called a stratum. So picture your evaluation dataset as a grid. One axis is task category — things like classification, summarization, extraction, generation, and reasoning. These represent different capabilities your model must perform. The other axis is difficulty level — easy, medium, hard. Five categories times three difficulty levels gives you fifteen strata, fifteen cells in the grid. Stratified sampling means you draw a controlled count of examples from every single cell, independently. Now, why does this matter? Because naive random sampling — where you just grab examples at random from a big pool — introduces two systemic risks. The first is category imbalance. If your raw pool has four hundred classification cases and only thirty generation cases, random sampling virtually guarantees that generation capability is under-tested. The second is difficulty skew. Annotators produce easy examples faster than hard ones, so random draws inflate your accuracy score artificially by over-representing the easy tail. Stratified sampling eliminates both risks by forcing every cell to contribute its fair share. Here's the analogy I like: imagine grading a math student using only multiplication questions and then claiming they understand mathematics. That's what flat random sampling does to your model. You walk away with a confidence number that collapses the moment production traffic shifts toward under-represented tasks. Now, the practical discipline. Every test case needs a minimum schema: the prompt text, the expected reference output, the task category label, the difficulty level, a unique case identifier for traceability, and a provenance field that records whether the example was human-authored or synthetically generated. In the labs you'll build this schema using a Python data validation library called Pydantic — think of Pydantic as a strict bouncer at a club. It checks every field, rejects malformed cases at the door, and guarantees that by the time a test case reaches your sampling pipeline, it conforms to the contract. The specific data structure you'll build is called EvalTestCase — that's the name you'll see in the labs, and it represents one single evaluation example with all its metadata. The non-negotiable rule is this: before publishing any dataset version, enforce a minimum count per stratum. A floor of about thirty examples per cell gives you enough statistical power that per-cell pass-rate estimates have tight confidence intervals. If any stratum falls below the floor, your sampling code must raise an error and refuse to produce the dataset. Silent under-population is how teams accidentally ship evaluations where the adversarial reasoning stratum has three examples and the summarization stratum has two hundred — and then they wonder why the overall score looks fine while production is on fire. Host: So stratified sampling is about forcing every combination of task category and difficulty level to contribute a minimum count, and refusing to publish if any stratum is starved. That mental model makes sense. But even a perfectly balanced dataset can be worthless if the examples have leaked into the model's training data. How do we catch that before it corrupts the scores? Expert: Right — this is where contamination detection comes in. Contamination is when evaluation examples, or near-paraphrases of them, show up in the model's training corpus. It's one of the most consequential quality failures in language model evaluation, because a contaminated benchmark can make a weaker model appear to outperform a stronger one. The model isn't reasoning — it's remembering. There are two complementary strategies. The first is called n-gram overlap analysis. The phrase "n-gram" just means a contiguous sequence of n words. An eight-gram is eight consecutive words. Here's the method: you break every evaluation prompt into overlapping eight-word windows, you break your reference training corpus into the same windows, and you compute what fraction of the evaluation prompt's n-grams also appear in the training n-grams. If the overlap ratio exceeds a threshold — typically about eighty percent — you flag that example as contaminated and exclude it. This works great when you have access to the training corpus. The second strategy is called membership inference, and it works even for third-party hosted models where you cannot see the training data. You query the hosted model's log-probability endpoint, which tells you how surprised the model is by each token of your evaluation prompt. Low surprise — suspiciously low perplexity relative to similar examples — is a fingerprint of memorization. The model has seen this text before. You compare each example's log-probability against a calibration distribution from the same stratum, and examples whose score is multiple standard deviations too confident get flagged. A production-grade pipeline runs both strategies. Now, critically, contaminated cases are not silently discarded. Their metadata gets annotated with the contamination ratio and the reason for exclusion. Every drop must be traceable, because when an audit team reviews your evaluation methodology six months later, they need to see why a case is missing, not just that it vanished. The other quality check that rides alongside contamination is inter-annotator agreement, often shortened to IAA. When humans label your cases, you want multiple annotators labeling the same example, and you compute a statistic like Cohen's kappa or Fleiss' kappa to measure how often they agree. Low-agreement cases — where even experts disagree on the correct answer — introduce label noise that corrupts your metrics in ways that are extremely hard to diagnose after the fact. The rule is: never include low-agreement examples without adjudication. Route them to a senior reviewer queue. Ambiguous ground truth is worse than missing ground truth, because it looks real but poisons every score it touches. Host: So we now have a dataset that's stratified, contamination-checked, and adjudicated for annotator agreement. That's a lot of engineering just to produce one dataset. But evaluation datasets aren't static — they evolve. Which brings us to versioning. If I run an evaluation today and a different one three months from now, how do I know the difference in scores came from the model and not from the dataset silently changing under me? Expert: This is the question that separates mature evaluation programs from amateur ones. Dataset versioning is the backbone of trustworthy evaluation, and it works differently from source code versioning. Code differences are small and human-readable — you can eyeball a diff. Dataset differences can span millions of lines, where a single duplicated row changes accuracy by a measurable amount, and no line-by-line comparison will ever reveal that. So you need a different approach, built on three layers. The first layer is content-addressable fingerprinting. "Content-addressable" means the identifier is derived from the content itself. The specific method is SHA-256, which is a cryptographic hash function that reads every byte of your dataset file and produces a fixed-length string. Two datasets with identical rows in identical order produce the exact same hash. Change a single character anywhere, and the hash becomes completely different. SHA-256 is the fingerprint that answers the question, "are these two datasets byte-identical?" with mathematical certainty. In the labs, you'll build a component called DatasetVersioner. It's the part of your pipeline responsible for computing that fingerprint, collecting structural metadata — row counts, category distributions, timestamps — and writing a version manifest, which is a small sidecar JSON file that sits next to the dataset and records everything about that snapshot. Crucially, DatasetVersioner reads the dataset in fixed-size chunks rather than loading the whole file into memory, because real evaluation datasets can reach multiple gigabytes. The second layer is the parent fingerprint chain. Every snapshot records the fingerprint of its predecessor, forming a linked version history analogous to how Git tracks commits. This lets you detect drift — category distribution shifts, row count changes, missing categories — between any two versions. Coverage gaps are especially dangerous. If your adversarial examples category silently disappears between version one and version two, your safety evaluation quietly becomes incomplete, and nothing in the raw score will tell you. The third layer is Git integration for tag-based reproducibility. You commit the dataset file and the version manifest together, then create an annotated Git tag — essentially a named bookmark with a message attached — using a namespace like "eval-dataset slash name slash v one point zero." The tag message encodes the fingerprint, row count, and category summary, so anyone browsing the repository can see what each dataset version contained without opening a file. Semantic versioning rules apply: a label correction that keeps the example set identical is a patch bump, adding examples is a minor bump, and changing the stratum schema requires a major bump. A pre-commit hook can enforce this automatically, rejecting mismatched version tags. And here's the payoff: months later, when a regulator or an internal audit team asks "what exact dataset produced this compliance report?", you run a reproduction script that checks out the tagged commit, re-runs the sampling pipeline with the recorded seed, and confirms byte-for-byte that the SHA-256 hash matches the stored artifact. If it doesn't match — often because a dependency upgrade silently changed random number generation — the script fails loudly and gives you a row-level diff. That diagnostic has saved teams weeks. Host: That's a clean picture. Fingerprint the bytes, chain each version to its parent, bind it to a Git tag, and reproduce on demand. But datasets don't just version forward — they go stale. Production traffic drifts. Users change behavior. A dataset curated six months ago may be testing query patterns that barely exist anymore. How do teams detect that and refresh the dataset without losing the ability to compare scores across versions? Expert: Dataset staleness is one of the most under-discussed problems in evaluation. Production traffic distributions shift continuously — seasonal effects, product launches, user behavior changes — and an evaluation dataset that was representative in January may over-represent abandoned query patterns by July. Left unchecked, you're essentially grading your model on a test it has long since outgrown. The detection mechanism uses a statistic called Jensen-Shannon divergence — usually just called JSD. Think of JSD as a number that measures how different two probability distributions are. Zero means identical. Higher values mean they've drifted apart. You compute JSD between your current production traffic distribution and your evaluation dataset distribution across multiple dimensions — task category proportions, prompt length distributions, entity frequency distributions. When any dimension's JSD crosses a configured threshold — a common default is around zero point one — you raise a staleness alert. JSD is preferred over a related measure called KL divergence because JSD is symmetric and always finite, whereas KL divergence breaks down when one distribution assigns zero probability to a category that appears in the other. Once staleness is detected, the refresh pipeline operates in what's called delta mode. This is important for preserving longitudinal comparability. The pipeline classifies every existing example as "retain," "retire," or "replace," and generates a candidate set of new examples to fill gaps in under-represented strata. Retired cases aren't deleted — they're moved to an archive partition tagged with the version in which they were retired. The target is to keep at least seventy percent of examples shared between consecutive versions, so score comparisons across versions remain meaningful. The orchestration layer is a scheduled workflow — an Airflow pipeline or a GitHub Actions job — that pulls the latest production traffic sample, runs the staleness detector, and when thresholds are breached, automatically opens a pull request with the candidate changeset. The dataset owner reviews, approves, and the new version gets tagged. This reduces the median time from staleness detection to dataset update from weeks of manual effort to hours of automated work. Now let me connect one more piece — synthetic data for privacy-constrained evaluation. When your source data contains personally identifiable information, protected health data, or confidential business content, regulations like GDPR, CCPA, and HIPAA often prohibit using the raw production data for evaluation. HIPAA in particular is the U.S. health data privacy law, and it imposes strict controls on protected health information. Naive anonymization distorts the linguistic patterns that models rely on, so evaluation results don't generalize. The production-grade answer is a tool called NeMo Safe Synthesizer — an NVIDIA framework that generates synthetic evaluation data with formal differential privacy guarantees. Differential privacy is a mathematical framework where a parameter called epsilon bounds how much any single real record can influence the synthetic output. Lower epsilon means stronger privacy. You configure epsilon based on your threat model, generate synthetic examples with provenance marked as synthetic, and attach a cryptographically signed privacy attestation document alongside the dataset tag. Auditors can then verify the privacy guarantees without ever touching the real source data. Host: That's the full production picture — sampling, versioning, staleness, privacy. Before we preview the labs, give me the production wisdom. If a listener remembers nothing else from this chapter, what are the two or three rules they must carry back to their team? Expert: Three rules. First: never use flat random sampling for evaluation datasets, and always enforce minimum-per-stratum floors before publishing. The number of teams that ship evaluations with three adversarial examples and two hundred easy classification cases is genuinely shocking. Set the floor, raise an error when any stratum is under-populated, and refuse to proceed. Second: pin all random seeds and dependency versions in your continuous integration environment. Reproducibility is a property of your whole pipeline, not just your data. A single unpinned dependency can change random number generation behavior and silently invalidate your byte-for-byte guarantee. When you re-run a version six months later and the SHA-256 fingerprint doesn't match, the root cause is almost always unpinned dependencies. Third: run contamination detection on a schedule, not just at curation time. Model providers update their training corpora without notification. A dataset that was clean against a hosted model in January may be contaminated against a March update, and your scores will quietly inflate. Schedule the contamination scan as a recurring job. And the one "never do this" that subsumes everything else: don't treat evaluation datasets as static artifacts. They are versioned, tested, continuously maintained engineering assets — not spreadsheets that rot in a shared drive. Host: Strong summary. Now let's preview the hands-on work. In the exercises, you'll practice each of these ideas directly. The first lab has you build a stratified evaluation dataset with minimum-per-stratum enforcement. The second implements dataset versioning and reproducible snapshots using content hashing and Git tags. The third builds contamination and leakage detection using n-gram overlap. The fourth is an automated dataset refresh pipeline driven by staleness detection. The fifth generates dataset cards — the structured documentation artifacts that record provenance, composition, and audit metadata. And the sixth builds an annotation pipeline with inter-annotator agreement checks and adjudication routing. Each lab has its own audio overview that goes deeper into the implementation. Take them in order; they build on each other. Host: To close out — you now understand three things you did not walk in with. First, you understand why stratified sampling across task categories and difficulty levels is non-negotiable for trustworthy evaluation, and why a minimum count per stratum must be enforced before any dataset is published. Second, you understand how content-addressable fingerprinting using SHA-256, paired with a parent-linked version chain and Git tag integration, produces evaluation datasets that are byte-for-byte reproducible years after the fact. Third, you understand how staleness detection with Jensen-Shannon divergence, contamination detection with n-gram overlap and membership inference, and privacy-compliant synthetic data using differential privacy all fit into a single production pipeline. You now have the depth to evaluate dataset curation approaches for your team's AI platform and to explain the trade-offs to your architects and stakeholders — this is exactly the kind of infrastructure skill that turns engineers who use AI tools into engineers who build the systems behind them. The chapter quiz will focus on Pydantic schema validation, the EvalTestCase structure, the role of SHA-256 in the DatasetVersioner component, and the regulatory constraints that drive synthetic data decisions — including which rules HIPAA imposes. Pay close attention to the decision point of when to use stratified sampling versus synthetic augmentation when a stratum is under-populated. In the next chapter, we move from the data to the judging. Chapter 2 covers language-model-as-judge evaluation — how to use a model like GPT-4o or Gemini to score the outputs of another model against a structured rubric. The datasets you curated here become the inputs that the judge evaluates. See you in Chapter 2.