Implement ADR staleness detection with drift alerts — lab audio overview

Host: Welcome back. You're in GenAI Architecture and Design Patterns — a course about the big structural choices behind production AI systems, and how to document and defend those choices over time. This chapter is about building an engine for Architecture Decision Records, which are short written documents where a team captures a technology choice and the reasoning behind it. Today's skill matters because decisions don't stay correct forever. Why does that hurt teams in practice? Expert: Picture a mid-sized fintech with maybe fifty engineers. Eighteen months ago, their architecture group picked a specific large language model from a specific vendor for customer support. They wrote it down, they justified it — the model was cheaper, latency was fine, accuracy was good enough. Everyone moved on. Now fast forward. The vendor has released three new model versions. A competitor dropped prices by sixty percent. Regulations around data residency have tightened. The original assumptions behind that decision — the cost numbers, the latency numbers, the vendor's data handling rules — are quietly no longer true. Nobody notices, because nobody is watching. The document just sits in a repository. That's what we call a stale decision. And the damage isn't theoretical. You're overpaying, you might be out of compliance, and your competitors are running circles around you because they're on newer, cheaper, better tools. What this exercise teaches you is how to build a system that watches every recorded decision, notices when its foundations have shifted, and raises an alarm before someone in a board meeting asks why you're still on last year's stack. This is exactly the kind of automation every serious GenAI platform team needs. Host: Okay, so we're turning documents into something living. In the previous exercise you built a validation framework — the part that pulls testable assumptions out of each decision record and checks them against real production numbers, things like actual cost, actual latency, actual accuracy. Now you're extending that. What exactly are we building here? Expert: You're building a staleness detector. Think of it as a smoke alarm for your architecture decisions. It reads every decision record your team has written. For each one, it looks at the assumptions that were captured — the model version, the expected cost per thousand requests, the latency target, maybe a regulatory constraint. Then it checks the current reality and asks one question: have any of these drifted far enough that this decision should be reconsidered? When the answer is yes, it produces an alert. And here's the key idea. The alert doesn't go into a private log or an email nobody reads. It goes out in a format that plugs directly into Alertmanager — that's the industry standard open-source tool that most engineering teams already use to route alerts to Slack, to PagerDuty, to on-call rotations. So your decision staleness shows up in the same place as a database outage or a memory leak. That's the conceptual shift. Architectural decisions become first-class operational signals. The same team that gets paged when a server dies now gets paged when a two-year-old model choice stops making economic sense. You're treating architecture as something that is monitored, not just something that is written. Host: That's a nice reframing. Before the listener starts, what's the part that tends to trip people up on this exercise? Expert: The hardest part is deciding what actually counts as stale. It's tempting to flag everything — any drift, any change, any new model release. If you do that, you create alert fatigue, and within a week your team will mute the channel and the whole system becomes useless. So you need tiers. A small cost increase is informational. A moderate drift, say cost has risen thirty percent above the original assumption, is a warning. A large drift, or a regulatory change, or a vendor deprecating the model you depend on — that's critical and should page someone. The tip is this: before you write any detection logic, sit down and list three or four severity levels with concrete thresholds. Thresholds in numbers, not adjectives. Then make sure each tier maps to a different routing rule in the alert output. That discipline is what separates a signal from noise. Host: Great guidance. So after this, you'll be able to take any written architecture decision, attach living assumptions to it, and have your monitoring system automatically tell your team when reality has moved away from those assumptions. This is a foundational capability for any platform team running GenAI in production — it turns architecture review from a yearly meeting into a continuous process, and it's exactly the kind of building block you can bring back to your own organization's design reviews. Next, in the final exercise of this sequence, you'll take everything the validation framework and the staleness detector have collected and roll it up into a scorecard — category-level statistics on how your decisions are actually performing, visualized on a dashboard your leadership can read at a glance. Thanks for listening, and good luck with the build.