Implement ROUGE-L overlap scoring — lab audio overview

Host: Welcome back. You're in GenAI Evaluation, Safety and Governance — a course about making sure the large language models your company deploys actually behave the way you expect in production. This chapter is about curating evaluation datasets — the test questions you use to judge whether a model is any good. And today's skill is one that separates teams who trust their numbers from teams who don't. Expert: Here's the scenario. Imagine you work on a platform team at a mid-sized insurance company. Maybe forty engineers, and you're the ones picking which hosted language model the whole company uses — something like OpenAI's GPT or Google's Gemini, accessed through a paid connection over the internet. To pick wisely, you built an evaluation set — a thousand carefully chosen question-and-answer pairs that represent real customer queries. You run the models against it, and one of them scores ninety-four percent. Fantastic, right? Except there's a quiet possibility nobody wants to talk about. What if that model already saw your test questions during its training? What if it isn't reasoning at all — it's just remembering? That's called test set contamination, and if it's happening, your evaluation numbers are fiction. Your leadership is making million-dollar vendor decisions on fiction. This exercise teaches you how to detect that contamination with math instead of guesswork. Host: So this is the second of three exercises in this sequence. In the previous one, you built something that asks the hosted model the evaluation questions and collects its answers. Now you're extending that. What exactly are we building today? Expert: Today you're building the measuring tape. In the previous exercise you got the model to produce answers to your test questions. Now you need to decide — how close is each answer to the one you expected? If the answer is suspiciously close, word-for-word, that's a red flag the model memorized it. The measuring tape you'll build is a well-known text-similarity score called ROUGE-L. The name comes from research on summarization — ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation, and the L stands for Longest. You don't need to memorize that. Think of it simply as an overlap score between zero and one. Zero means the two pieces of text share nothing in common. One means they're identical. And importantly — you're not using an outside library for this. You're writing the algorithm yourself, so you truly understand what the number means. Then, on top of that single-pair scorer, you'll build a batch scorer — a piece of code that runs the comparison across your whole evaluation dataset and reports what fraction of test cases look contaminated. Host: Okay, let's slow down on the key idea. What's the core insight behind this overlap score? Why does it work? Expert: The insight is called longest common subsequence. Imagine you have two sentences. The model's answer, and the expected answer. Walk through both, left to right, and look for the longest sequence of words that appears in both — in the same order, but not necessarily back-to-back. So if the expected answer is "the cat sat on the mat" and the model says "the big cat sat lazily on the mat," the longest shared sequence is "the cat sat on the mat" — six words, in order, even though the model added words in between. That shared length, compared against the length of both sentences, gives you the overlap score. Here's why that matters for contamination. A model that genuinely reasoned about a question will produce an answer that's semantically right but worded differently — different sentence structure, different vocabulary. A model that memorized the answer will reproduce long stretches of the original text word-for-word, because that's what memorization looks like. So a high overlap score is a fingerprint of memorization. Then your batch scorer runs this across every test case, counts how many cross a suspicious threshold — say, point eight or higher — and reports the contamination rate as a percentage of your dataset. That's the number you bring to your team. Host: Before someone starts coding — what's the thing that trips people up here? Expert: Two things. First, the algorithm itself. Finding the longest shared sequence between two sequences is a classic problem, and if you try to solve it the obvious way — by checking every possible combination — it explodes on long texts. The standard approach uses a grid, where you compare each word in one text against each word in the other and fill in a table of running totals. It's efficient, but the logic is subtle. Take your time. Draw the grid on paper for a tiny example first — three words against three words — and trace through it by hand before you write any code. Second trap: how you prepare the text before comparing. If one answer has a trailing period and the other doesn't, or one is capitalized and the other isn't, a naive comparison will count them as different and tank your score. So decide up front — lowercase everything, strip punctuation, split on whitespace — and apply the exact same preparation to both sides. Be consistent, or your numbers will lie to you in a very confusing way. Host: Perfect. So wrapping up — what will someone be able to do after this, and what's coming next? Expert: After this exercise, you'll be able to take any two pieces of text and produce a defensible, numerical measure of how much they overlap — and more importantly, you'll understand exactly what that number means because you built it yourself. You'll be able to scan an entire evaluation dataset and tell your team, with evidence, "here is the percentage of our test cases that look contaminated on this model." That's a foundational building block your team can reuse anywhere you need to judge text similarity — not just contamination, but also things like measuring whether two customer complaints are near-duplicates, or whether a generated summary captured the source material. In the next and final exercise of this sequence, you'll take the contamination scores you produce today and build the workflow around them — a system that isolates the suspicious test cases, records the evidence, and generates a report your team can act on. So today is the measurement. Next is the response. Go build it — and thanks for listening.