Create automated compliance checks — lab audio overview

Host: Welcome back. You're in GenAI Evaluation, Safety and Governance — a course about making sure the AI systems your team ships are trustworthy, auditable, and legal to operate. This chapter is on the European Union AI Act — that's the new law in Europe that regulates artificial intelligence systems, and the penalties for breaking it are now active. Today's skill is automating the checks that prove you're following that law. Expert: Picture this. You're on a mid-sized engineering team at a company that sells a customer service chatbot into Europe. Maybe eighty engineers, a handful of AI products. Your legal team has just told you that under the new European law, you need to prove — on demand, with evidence — that your system meets certain technical requirements. Things like: do you have proper documentation? Are you keeping logs of how the model is used? Is your monitoring system actually running? Now, someone could check all this manually once a quarter, dig through folders, ping services by hand, write up a report. But the law is active now, the fines are real — we're talking up to seven percent of global annual revenue for the worst violations — and manual checks drift out of date the moment they're done. What your team actually needs is a system that checks itself, continuously, and can produce a human-readable compliance report at any moment. That's what you're building today. Host: Good, so the pressure is real. Before we get into today's build — in the previous exercise the student built something specific. How does today connect? Expert: Right. In the previous exercise, you built what we called a compliance tracker. Think of it as a calendar of the four key enforcement dates in the European law — February 2025, August 2025, February 2026, and August 2026 — and under each date, a checklist of the specific things your organization has to have in place by then. That's the static picture. It tells you what needs to be true. What it doesn't tell you is whether those things are actually true right now, today, in your running system. That's the gap you're closing in this exercise. You're going to build an automated compliance checker. In plain language, that's a piece of code that holds a list of small verification routines — each one tied to a specific legal requirement — and can run them all on a schedule, collect the results with timestamps, and then turn those raw pass-fail results into a readable narrative that a compliance officer or an auditor can actually understand. The key idea — the aha — is this: compliance isn't a document, it's a live signal. You're turning a checklist into a heartbeat. And the narrative piece uses the Anthropic API — that's the service from the company Anthropic that lets you call their Claude language model from code — to translate dry technical results into clear prose. Host: Let's talk about the specific checks. What's actually getting verified, and then — what trips people up? Expert: The exercise has you wire up three concrete kinds of checks, and they're deliberately simple so you focus on the plumbing, not the content. First, a documentation check — does a required file actually exist on disk? Second, a log check — is there evidence that the system is writing usage logs? Third, a health check — is a specific service endpoint responding? Each of these is just a small routine that returns pass or fail with a short reason. Now, the tricky part. The pattern here is called a registry. You don't hard-code the list of checks inside the checker itself. Instead, you register each check with the checker, giving it a name and tying it to a specific compliance requirement from the previous exercise. This feels like extra ceremony the first time you do it, and people are tempted to just write one big routine with everything inline. Don't. The whole point is that next month, when legal adds a new requirement, your teammate should be able to add one new check and register it — without touching the checker itself. The second trap is the language model call. When you ask Claude to write the narrative, feed it the structured results — the raw pass-fails with timestamps — not a pre-summarized version. The model is much better at summarizing clean data than at cleaning up your summary. And wrap that call in error handling, because if the network hiccups, your compliance report shouldn't just crash — it should fall back to the raw results and flag that the narrative is missing. Host: Great. Let's close out — what does the student walk away able to do, and where does this lead? Expert: After this exercise, you'll be able to take a set of legal or policy requirements and turn them into a living, self-checking system that runs on a schedule, records what it found, and produces a readable report for non-engineers. That's a building block your team can drop into almost any regulated product — not just European AI Act work, but internal policy enforcement, security reviews, vendor compliance, anything where the question "are we still doing the thing we said we'd do" needs a machine-checkable answer. In the next exercise, you'll take the results this checker produces and build the last piece — a tool that identifies which requirements are failing or partially met, and ranks them by how close the enforcement deadline is, so your team knows exactly what to fix first. Thanks for listening, and good luck with the build.