GenAI ADR Engine — chapter audio overview

Podcast Script: GenAI ADR Engine Host: Welcome to Chapter 1 of 26 in *GenAI Architecture and Design Patterns* — "The GenAI ADR Engine." Picture this: you're six months into running a production GenAI system. Summarization costs have tripled. Someone on the team swapped the embedding model in a commit last quarter. Nobody remembers why you chose retrieval-augmented generation over fine-tuning for the support bot. Your architecture has quietly drifted, and the Slack threads that justified each choice are long gone. That's the problem this chapter solves. Your organization invested in this training so you can build the infrastructure that keeps GenAI decisions visible, validated, and governed — not just implemented. This is a core competency for any team shipping production AI systems, the kind of depth that turns an engineer who uses AI tools into one who builds the governance layer behind them. You'll practice across six hands-on labs, but first, let's build the mental model. We'll walk through five big ideas together — a typed decision record schema, multi-provider model comparison, telemetry-driven validation, decision dependency graphs, and governance dashboards. By the end, you'll understand how to turn architecture documentation from a forgotten wiki page into a living control plane. Host: Let's start with the core concept. What is an Architecture Decision Record, and why does GenAI need its own flavor of it? Expert: Good place to start. An Architecture Decision Record — ADR for short — is a structured document that captures a single architecture decision. It records the context you were in, the options you considered, the rationale for the choice, and the consequences that follow. The format has existed in traditional software for over a decade, usually as a short prose document sitting in a folder in the codebase. The problem is, traditional ADRs were designed for stable choices — which database to use, which message broker to pick. Those get reviewed once every few years. GenAI decisions decay in months. A model you chose in January might be superseded by something 40% cheaper by April. So we need a record format that isn't just prose — it has to be machine-readable, so automated systems can continuously check whether each decision still holds. The chapter builds around a concept called a decision taxonomy. Think of a taxonomy as a controlled vocabulary — a short, fixed list of categories every record must fit into. For GenAI systems, three categories dominate. Model selection asks which foundation model powers a given capability — something like GPT-4, Claude, or a self-hosted open-weight model. This decision depends on token cost, latency percentiles, quality scores, context window size, and vendor lock-in risk. Hosting strategy asks where the model actually runs — fully managed APIs from providers, cloud-hosted endpoints, or self-hosted servers on your own GPU infrastructure. Each trades cost control against operational burden. The third category is the RAG-versus-fine-tuning decision. RAG stands for retrieval-augmented generation — it keeps the base model frozen and injects relevant context at query time. Fine-tuning bakes knowledge directly into the model's weights during a training phase. The right answer depends on how fresh your data needs to be, your latency budget, and whether you even have training infrastructure available. Now, the schema itself. Every record carries a handful of structured fields: a unique identifier, a category drawn from that fixed taxonomy, a context dictionary with category-specific metadata, a criteria-weights mapping that captures how much you care about cost versus latency versus quality, and a list of dependencies pointing to other records this one builds on. There's also a status field — proposed, accepted, superseded, deprecated, or rejected — that models the full lifecycle of a decision. Two implementation details matter. First, the record includes a validation routine that refuses to save any record missing required fields. If you try to create a model-selection record without a provider name, it throws an error immediately. We build this in Python, using dataclasses for the structure and typed enumerations for the category and status. Some teams reach for Pydantic — that's a Python library for data validation that gives you the same structural enforcement with richer error messages. Either way, the idea is that a malformed record never makes it into the registry. Second, every record carries an expiry date. A six-month-old model decision is probably stale, full stop. That expiry date is what later lets the governance layer automatically flag decisions for review — rather than relying on engineers to remember which choices need revisiting. Host: Okay — so we have a structured record format with typed categories and expiry dates built in. The next question is: once you have that schema, how do you actually make a model selection in a way that isn't just "my favorite vendor"? Expert: This is where the weighted criteria matrix comes in, and it's one of the most valuable patterns in the chapter. A weighted criteria matrix is a scoring framework where you list your evaluation dimensions — cost, latency, quality — assign each dimension a weight reflecting how much you care about it, score every candidate on every dimension, and compute a composite score. The weights must sum to one, which forces the team to confront trade-offs explicitly rather than pretending everything matters equally. Here's the workflow. For each candidate model, you gather what we call a capability profile — a uniform record of what the model offers. This profile normalizes across providers, so an Anthropic model and an OpenAI model expose identical fields. It captures three things. A cost profile with per-token pricing for input and output. A latency profile with both median and tail latency under realistic concurrent load — not single-request benchmarks from a blog post. And a quality profile with benchmark scores tied to specific evaluation tasks. That last point is important. A quality score of 0.87 means nothing without knowing whether it represents a coding benchmark or a summarization benchmark. So every quality score in the profile is tagged with the task name, the dataset, and the metric. This traceability is what makes the decision auditable six months later. Once you have profiles for every candidate, the scoring engine normalizes each dimension to a zero-to-one scale. For cost and latency, the logic is inverted — the cheapest model scores highest, the fastest model scores highest. For quality, higher raw scores stay higher. Then the engine applies the weights and produces a ranked list. The output feeds directly into the ADR record. The top-ranked model becomes the decision, and — this is critical — every other candidate gets recorded in an "alternatives considered" field along with its scores. That means a future engineer who asks "why didn't we pick Claude?" has a concrete answer with numbers, not a shrug. A production gotcha worth calling out: don't hardcode provider-specific dimensions into your scoring framework. If your matrix has a dimension named after a specific vendor's rate-limit tier, you've locked yourself to that vendor. Use provider-agnostic dimensions like throughput tokens per second, cost per million input tokens, or context window size — and let adapters translate each provider's terminology onto those universal dimensions. When a new provider shows up next quarter, your scoring framework doesn't need a rewrite. One more principle. The weights themselves are an architecture decision — they encode what your organization values. A cost-sensitive startup might weight cost at 0.5 and quality at 0.3. A latency-critical trading platform might invert those completely. Documenting those weights inside the record makes trade-offs explicit and auditable. When stakeholders disagree about model choice, the conversation shifts from "which model is better" to "what weights reflect our priorities" — which is a much more productive discussion. One more practical consideration. Some dimensions aren't really scoring dimensions — they're hard filters. If a model lacks function-calling support and your architecture requires agents, no weighting can rescue it. These hard constraints eliminate candidates before the weighted matrix even runs, keeping the scoring focused on genuinely viable options. Host: So the matrix gives us a defensible, reproducible decision. But a decision that was right six months ago might be wrong today. How do we keep it honest as production reality shifts? Expert: This is where telemetry validation comes in, and it's the layer that separates living documentation from a dusty archive. The idea is simple: every record contains assumptions — assumptions about cost, latency, and quality. We treat those assumptions as first-class, testable predicates and continuously compare them against live production metrics. Each assumption has three parts. A metric name that names the thing we're measuring — something like p95 latency in milliseconds or cost per request. A condition like less-than, greater-than, or within-a-range. A threshold value. And — this matters — a tolerance band, usually around ten percent. Without the tolerance band, every normal fluctuation fires a false alarm, and teams quickly learn to ignore the system. With it, only real drift triggers attention. We also track what we call consecutive violations. A single bad data point doesn't mean anything — networks hiccup, providers burp. But three or four consecutive validation cycles showing the same violation signal that something real has changed. Only at that point does the assumption get marked stale. The validation pipeline itself runs on a schedule — maybe every fifteen minutes. It loads active records, pulls the corresponding metrics from your telemetry backend (something like Prometheus or Datadog), and evaluates each assumption. Every assumption lands in one of four states. Confirmed, if it's within threshold. Degraded, if it's past threshold but inside the tolerance band. Violated, if it's outside tolerance. Or unknown, if telemetry is temporarily unavailable. That middle state — degraded — is the early warning. It tells you a decision is trending toward staleness before it fully breaks. The governance dashboard surfaces degraded records as yellow, not red, so teams can investigate without panicking. Here's where it connects to everything else. When a record flips to stale, the system doesn't just send a notification. It can automatically re-run the weighted criteria matrix with current telemetry values. If the matrix now recommends a different model, the governance layer creates a reconsideration ticket with both the original scores and the updated scores side by side. The reviewing architect gets concrete data, not a vague "please review." A few deployment lessons. First, calibrate the validation cadence to your metric aggregation window. If your metrics backend aggregates in five-minute buckets, validating every minute gives you noisy partial-window results. A fifteen-minute validation cycle on five-minute aggregates produces stable readings. Second, tune the consecutive-violation threshold per category. Quality metrics are noisier by nature — allow five or seven consecutive violations before flagging. Cost is more stable and compounds fast — flag after two or three. Third, don't use one global tolerance threshold for everything. A fifteen percent cost deviation on an overnight batch summarization job is fine. A fifteen percent latency deviation on a real-time chat endpoint is a critical regression. Configure tolerances per category and per dimension. Host: Okay — now we can keep decisions honest against production reality. But decisions don't live in isolation. One choice constrains another. How does the engine model that? Expert: This is the decision dependency graph, and it's the piece that turns a flat record registry into a reasoning system. The core observation is that architecture decisions in GenAI form chains. A hosting strategy decision — say, committing to self-hosted inference — constrains which models you can even consider, because you're limited to open-weight models. That model choice then constrains your prompt architecture, because different models have different system prompt conventions and tool-calling formats. The prompt architecture constrains your guardrail strategy, which constrains your retrieval pipeline's context budget. Change any one link, and everything downstream may silently become invalid. We model this as a directed acyclic graph. Each record is a node. Each dependency is a directed edge pointing from the upstream decision to the downstream one. The word "acyclic" is the key constraint — circular dependencies mean two decisions each claim to depend on the other, which makes impact analysis impossible because the traversal never terminates. The engine refuses to insert any edge that would create a cycle, and if you try, it tells you exactly which path forms the loop. The edges themselves are typed, and the types matter. An edge can be a "constrains" relationship — a hard runtime dependency. It can be an "enables" relationship — softer, meaning the upstream decision made the downstream one possible but doesn't strictly require it. Or it can be a "conflicts" relationship — two decisions that are mutually exclusive. The governance layer flags it as a violation when both accidentally end up in accepted status. The primary query we run on this graph is called impact analysis. Given that a specific record is being reconsidered, which other records are potentially affected? The engine does a breadth-first walk following outgoing edges from the change point, collecting every reachable downstream record. The output includes the list of affected decisions, the depth — how many hops from the change — and a severity level derived from edge type and depth. A "constrains" edge at depth one is critical. An "enables" edge at depth three is low severity. Depth matters for prioritization. A decision one hop away needs immediate review. A decision three hops away can probably wait for the next quarterly cycle. The graph also supports reverse impact analysis. When an engineer drafts a brand-new record, the system walks backward through incoming edges to find every upstream decision this new choice depends on, then checks their status. If any upstream dependency is in a review-required or expired state, the engine warns the engineer before accepting the new record. That prevents the common failure mode of silently building on an invalidated foundation. One production pattern worth naming: version the graph alongside the records themselves. When an edge is added or removed, record the change in an append-only audit log with the actor, the timestamp, and a short rationale. This log tells auditors not just what decisions were made, but how the team's understanding of the relationships evolved. For post-incident reviews, being able to reconstruct the graph state at the moment of the incident is enormously valuable. Host: So we can now document, score, validate, and trace cascading impacts. The last piece: how do you keep a whole organization actually doing this work over time? Expert: That's the governance dashboard, and it's the control plane that ties everything together. The dashboard answers three questions continuously. What decisions exist in production but aren't documented? Which documented decisions are overdue for review? And who must act, by when? Start with the first — detecting undocumented decisions. The most dangerous decision is the one nobody wrote down. A developer quietly swaps embedding models in a commit, and suddenly retrieval recall shifts without any record capturing the rationale. The detector cross-references two inventories. The declared inventory is what the record registry knows about. The observed inventory is what's actually running — distinct model identifiers in deployment configs, endpoint URLs in your infrastructure-as-code files, provider SDK imports in your source code, and inference-related charges in your cloud billing. Anything in the observed set that doesn't match a record in the declared set is flagged as a governance gap, with a severity level tied to whether the resource is running in production or just staging. The dashboard also enforces review workflows. Every record carries two temporal fields — a next-review date and a hard expiry date. For GenAI decisions, these cadences are aggressive. Model selection records get a 90-day review cycle because provider pricing and capabilities shift quarterly. Hosting strategy records can stretch to 180 days because infrastructure migrations are slower and more expensive. When a record passes its review date, the workflow engine creates a review task and assigns it to the decision owner — typically round-robin across a team roster. If nobody completes the review within 14 days, the system escalates to the tech lead. If the hard expiry date passes without resolution, the record transitions to expired, and — here's the powerful part — every downstream record that depends on it automatically gets frozen. Dependent teams can't merge new decisions built on an expired foundation until the foundation is refreshed. The dashboard aggregates all of this into a health score per decision category. Critical gaps cost heavily, expired records cost even more, review-due items cost less. A category with a health score of 0.3 is in serious trouble. A category at 0.95 is well-governed. And this score feeds back into a recommendation engine — a system that suggests architectural options for new decisions based on historical outcomes, weighted by the governance health of those past decisions. That creates a virtuous cycle: teams that maintain their records get better recommendations, which incentivizes maintaining them. One integration worth highlighting. The engine plugs into your continuous-integration pipeline as a merge gate. If a pull request touches model configuration files, retrieval pipeline definitions, or hosting manifests, the CI check requires a linked, accepted record before the merge can proceed. That single integration turns the whole system from voluntary documentation into an actual engineering control. Without it, the governance layer is toothless. Host: This is a lot of moving parts. If someone only remembers three things from this chapter, what should they be? Expert: Three lessons stand out. First — if you remember nothing else: attach quantitative evidence to every decision. A record that says "we chose GPT-4 for summarization" without the latency numbers, the cost projections, and the quality scores is not an engineering artifact — it's an opinion. Store the full weighted criteria matrix inside the record's evidence field so future reviewers can recalculate the outcome if the weights shift. Second — and this is the top thing *not* to do — never store evidence as unstructured prose paragraphs. When benchmark results and cost calculations live inside free-text context fields, no automated system can parse, compare, or re-evaluate them. Use typed, structured fields for evidence. Prose belongs in the rationale field for human readers. Evidence belongs in queryable structures. Third, require dependency declarations at record-creation time. Retroactively mapping dependencies between dozens of existing records is an error-prone archaeology exercise teams perpetually defer. Make the upstream-dependencies field mandatory in your schema validation. If an engineer can't articulate which existing decisions their new choice depends on, that's a signal they don't fully understand the system's decision landscape yet — and the record shouldn't be approved until those relationships are identified. One final warning: don't conflate ADR governance with organizational approval chains. The engine validates that decisions are documented, scored, and aligned with production telemetry. It does not replace architecture review boards or security reviews. Keep it focused on technical decision integrity, and integrate with external approval systems through webhook events rather than embedding approval logic directly into the schema. The moment you start encoding your company's reporting lines inside record fields, the whole system becomes brittle every time the org chart changes. Host: That's a clean mental model to carry into the labs. Speaking of which — you'll practice each of these ideas hands-on across six exercises. Lab one walks you through building the schema and decision taxonomy itself — defining the core categories as typed enumerations and the record structure as a validated dataclass. Lab two is the multi-provider comparison workflow, producing scored decision matrices across cost, latency, and quality. Lab three wires up telemetry validation against mock production metrics. Lab four builds the decision dependency graph with cycle detection and impact analysis. Lab five implements the recommendation engine that uses historical outcomes. And lab six assembles the full governance dashboard and compliance audit. Each lab has its own audio overview that goes deeper. To close — you now understand three things. First, why GenAI decisions demand a typed, structured ADR format rather than traditional prose — because they expire in months, carry quantitative trade-offs across multiple dimensions, and form cascading dependency chains. Second, how a weighted criteria matrix turns model selection from opinion into auditable engineering, and how telemetry validation keeps those decisions honest against production reality. Third, how decision dependency graphs and governance dashboards scale this practice from one team to an entire organization. You now have the depth to lead the architecture decision process on your team's GenAI initiatives — to stand up the record format, define the taxonomy that fits your domain, and explain the trade-offs to both engineers and stakeholders. This is the infrastructure that separates teams who can confidently evolve their GenAI stack from teams afraid to change anything. The chapter quiz will focus on the decision category taxonomy — especially how to classify choices involving GPT-4, Claude, and self-hosted providers — how the criteria matrix uses weights, and how data validation with typed schemas like Pydantic enforces record integrity. Pay close attention to which provider attributes belong in which category fields, and how the enumerated decision category interacts with the required context keys. Next, in Chapter 2, we extend this foundation into a reference architecture registry — a system that turns validated decisions into reusable architecture patterns that teams across your organization can adopt. See you there.