Implement decision option scoring with weighted criteria matrix — lab audio overview

Host: Welcome back to GenAI Architecture and Design Patterns. You're in the chapter on the GenAI ADR Engine — and ADR stands for Architecture Decision Record, which is basically a written memo that captures a significant technical choice, why you made it, and what you ruled out. This chapter is about building a system that manages those memos automatically. Why does that matter on a Monday morning? Expert: Picture a mid-sized fintech company, maybe forty engineers split across three teams. One team picks a large language model from one vendor because it's cheapest per request. Another team picks a different model because it scored better on their internal tests. A third team is arguing about whether to use retrieval — which means pulling relevant documents from a database at question time — versus fine-tuning, which means retraining the model on your own data. Six months later nobody remembers why any of these choices were made. The vendor bills are climbing, a security review is coming, and leadership asks the architect a simple question: "Which model should we standardize on?" Without a structured way to score options against real criteria — cost, latency, accuracy, compliance — that conversation turns into opinions and LinkedIn posts. What you're about to build is the piece that replaces opinion with a number. A transparent, reproducible way to compare options, so when someone asks "why did we pick this model?" there's a real answer on file. Host: Good. So in the previous exercise you built the skeleton — the typed shapes that describe what a decision record actually looks like, including its categories, its lifecycle status, and the web endpoints that create and retrieve them. Now you're extending that skeleton with the thing that makes decisions defensible. What exactly is the listener building here? Expert: You're building three things that work together. First, a structured representation of a single option on the table — say, "use the Claude model" or "use an open-source model hosted ourselves." Each option carries its name, a short description, and most importantly, a set of scores across different criteria — things like cost per thousand requests, average response time, accuracy on your test set, and regulatory fit. Second, you're building a weighted criteria matrix — and that's the key phrase, so let me unpack it. Imagine a spreadsheet where the rows are your options and the columns are your criteria. Each cell holds a raw score. But not every criterion matters equally — for a healthcare app, compliance might be worth three times what latency is worth. So each column gets a weight, the scores get normalized onto a common scale so that "dollars" and "milliseconds" and "accuracy percent" can be compared fairly, and then everything collapses into a single ranked list. Third, you're building small helpers that generate a human-readable comparison report and that validate the evidence — meaning, they check that every score has a source behind it, not just a number someone made up in a meeting. Host: That matrix idea is where I want to slow down. What's the conceptual "aha" the listener needs before they open the editor? Expert: The "aha" is this: raw scores are lies until you normalize them. If option A costs two cents and option B costs twenty cents, and option A has latency of eight hundred milliseconds and option B has latency of two hundred, you cannot just add those numbers together. The units are different and the directions are different — lower cost is better, but higher accuracy is better. Normalization means putting every criterion onto the same zero-to-one scale where one always means "best on this criterion" and zero always means "worst." Once every score is on that common scale, you multiply each one by its weight, add them up, and you get a single comparable number per option. That's the ranking. The magic isn't the math — the math is just averages. The magic is that the weights are explicit and written down. When the compliance officer says "we should have weighted regulatory fit more heavily," they can change one number and rerun the comparison. That's the whole point of this exercise — making the decision process auditable, not just the decision itself. Host: Before we let them loose, what's the one thing that's going to trip people up? Expert: Direction of scoring. Some criteria are "higher is better" — accuracy, throughput, reliability. Some are "lower is better" — cost, latency, risk. If you normalize them all the same way, your matrix will quietly rank the most expensive, slowest option as the best, and you won't notice until someone asks why. So before you start, decide for every criterion whether bigger numbers are good or bad, and make sure your normalization step flips the lower-is-better ones. The second snag is weights that don't add up to one — if they sum to more or less than one, your final scores become hard to interpret and hard to compare across different decisions. Check that sum early. Host: Perfect. So after this exercise, what can the listener actually do? Expert: You'll be able to take any architectural decision — which model to use, whether to host it yourself or call a hosted service, whether to go with retrieval or fine-tuning — and produce a ranked, weighted, evidence-backed comparison that a review board can actually defend. For your team, this becomes the reusable scoring engine behind every significant GenAI choice your organization makes this year. And in the next exercise, you'll add the memory layer — a storage system that keeps every version of every decision, tracks when one decision replaces another, and builds the full history chain so nothing ever gets lost. Thanks for listening, and good luck in the editor.