Create governance dashboards that surface undocumented decisions and enforce review workflows with expiry tracking

When you ship GenAI architecture without an active governance layer, undocumented model swaps quietly displace approved ones, expired ADRs keep anchoring downstream choices nobody has revalidated, and the cost surfaces months later as a regression no one can trace back to a decision — because no decision was ever recorded. By the end of this lesson, you'll be able to design a governance dashboard that detects undocumented decisions from telemetry gaps, enforces review workflows with expiry tracking, and aggregates per-category health scores that feed back into the ADRRecommendationEngine so poorly governed decisions contribute less to future recommendations.

• GovernanceGap: a record representing an undocumented architecture decision detected when production telemetry surfaces a model, endpoint, or retrieval strategy that has no matching ADR in the registry; carries severity, environment, first-seen timestamp, and suggested category for triage.
• ADRRecommendationEngine: the downstream component (introduced in the lab objective) that ranks historical architecture decisions when suggesting new ones; consumes per-category health scores from the dashboard so poorly governed decisions contribute less to future recommendations via a similarity_score * governance_weight ranking term.
• CategoryHealth: a per-decision-category aggregate score (0.0–1.0) produced by the GovernanceDashboard by combining undocumented-decision counts, overdue reviews, and expired ADRs within that category; it is the unit of governance signal that flows back into the recommendation engine.

Connecting Governance to the Recommendation Engine

The health scores produced by GovernanceDashboard feed directly into the ADRRecommendationEngine from the lab objective. When the recommendation engine retrieves historical outcomes to suggest optimal architecture choices, it weights those outcomes by their governance health score. A past decision with a health score of 0.3 (indicating expired reviews and undocumented related decisions) contributes less to future recommendations than a well-governed decision scoring 0.95. This creates a virtuous cycle: teams that maintain their ADRs get better recommendations, which incentivizes governance compliance.

The key integration point is the similarity_score * governance_weight multiplication in the recommendation engine's ranking function. By making governance health a first-class input to automated recommendations, you transform the dashboard from a compliance checkbox into a system that actively improves decision quality over time. Expired model selection ADRs do not just trigger Slack notifications—they degrade the recommendation engine's confidence in suggesting similar models, forcing teams to re-evaluate before the engine will confidently recommend that path again.

This tight coupling between ADR governance, telemetry validation, and the recommendation engine is what distinguishes a production-grade ADR system from a documentation template. The dashboard is not a view layer—it is the control plane for your organization's GenAI architecture decisions.

Now that you have seen how governance health scores flow into the ADRRecommendationEngine, the next step is making those scores concrete: first detecting undocumented decisions from telemetry gaps, then aggregating them into the per-category health signal the recommendation engine consumes.

The most dangerous architecture decisions are the ones nobody wrote down. A developer quietly switches from text-embedding-ada-002 to text-embedding-3-small in a single commit, and your RAG pipeline's recall changes without any ADR capturing the rationale. The detector cross-references two inventories: the declared set (ADR subjects in the registry) and the observed set (model identifiers, endpoints, and retrieval strategies seen in production telemetry). Anything observed without a matching ADR becomes a GovernanceGap.

Code snippetpython
1from dataclasses import dataclass, field
2from datetime import datetime
3from enum import Enum
4from typing import Optional
5
6class GapSeverity(Enum):
7    CRITICAL = "critical"   # production, no ADR
8    WARNING = "warning"     # non-production, no ADR
9
10@dataclass
11class GovernanceGap:
12    observed_resource: str
13    environment: str
14    first_seen: datetime
15    severity: GapSeverity
16    suggested_category: str
17
18@dataclass
19class UndocumentedDecisionDetector:
20    adr_subjects: dict[str, str] = field(default_factory=dict)
21    similarity_threshold: float = 0.85
22
23    def detect_gaps(self, observations: list[dict]) -> list[GovernanceGap]:
24        gaps: list[GovernanceGap] = []
25        for obs in observations:
26            if self._match_adr(obs["resource_identifier"]) is None:
27                sev = (GapSeverity.CRITICAL
28                       if obs["environment"] == "production"
29                       else GapSeverity.WARNING)
30                gaps.append(GovernanceGap(
31                    observed_resource=obs["resource_identifier"],
32                    environment=obs["environment"],
33                    first_seen=datetime.fromisoformat(obs["timestamp"]),
34                    severity=sev,
35                    suggested_category=obs.get("category", "unknown"),
36                ))
37        return gaps
38
39    def _match_adr(self, resource: str) -> Optional[str]:
40        observed = set(resource.lower().replace("-", " ").replace("_", " ").split())
41        for subject, adr_id in self.adr_subjects.items():
42            tokens = set(subject.split())
43            denom = max(len(observed), len(tokens), 1)
44            if len(observed & tokens) / denom >= self.similarity_threshold:
45                return adr_id
46        return None

Detection is only half the problem. The GovernanceDashboard rolls gaps up into a CategoryHealth score (0.0–1.0) per decision category, combining undocumented-decision counts with overdue reviews and expired ADRs, then exposes the similarity_score * governance_weight term the recommendation engine uses for ranking.

Code snippetpython
1@dataclass
2class GovernanceDashboard:
3    detector: UndocumentedDecisionDetector
4
5    def category_health(self, category: str, observations: list[dict],
6                        overdue_reviews: int, expired_adrs: int) -> float:
7        gaps = [g for g in self.detector.detect_gaps(observations)
8                if g.suggested_category == category]
9        penalty = len(gaps) + overdue_reviews + expired_adrs
10        return max(0.0, 1.0 - 0.1 * penalty)
11
12    def weighted_similarity(self, similarity_score: float, health: float) -> float:
13        return similarity_score * health  # feeds ADRRecommendationEngine ranking

A clean category returns 1.0; one with expired reviews and undocumented swaps drops toward 0.3, dragging down any recommendation that leans on it. Verify by feeding a telemetry batch containing one production resource with no matching ADR: detect_gaps should return exactly one CRITICAL GovernanceGap, and category_health for that category should fall below 1.0 while a fully documented category stays at 1.0.

Do's

1. ✓Do cross-reference observed telemetry against declared ADR subjects using UndocumentedDecisionDetector.detect_gaps — quiet substitutions like a text-embedding-ada-002 → text-embedding-3-small swap never appear in a code review, only in telemetry; without this two-inventory diff, the RAG recall regression surfaces months later with no decision trail to audit.
2. ✓Do distinguish GapSeverity.CRITICAL (production) from GapSeverity.WARNING (non-production) at gap-creation time inside detect_gaps — severity is set once per observation so that category_health and dashboard displays can prioritize undocumented production resources without re-examining environment data downstream.
3. ✓Do pipe weighted_similarity(similarity_score, health) — the similarity_score * health product — into ADRRecommendationEngine ranking — this is how expired ADRs and overdue reviews penalize future recommendations automatically; a category with penalty >= 7 collapses to health = 0.3, dragging any recommendation that leans on it far enough down to redirect architects toward better-governed alternatives.

Don'ts

1. ✗Don't lower similarity_threshold below 0.85 in _match_adr — the denominator is max(len(observed), len(tokens), 1), so a short ADR subject like "embedding model" scores high overlap against nearly any two-token resource identifier, suppressing real gaps and producing a falsely clean category_health score.
2. ✗Don't omit overdue_reviews or expired_adrs from the category_health penalty — counting only undocumented-decision gaps lets an ADR that exists but has never been revalidated return health = 1.0, which feeds the ADRRecommendationEngine a clean governance signal for a category that is functionally unreviewed.
3. ✗Don't allow suggested_category to default to "unknown" without a resolution strategy — category_health filters gaps by exact category name, so any gap filed under "unknown" is invisible to every named-category penalty calculation and silently exempts undocumented decisions from dragging down weighted_similarity in the recommendation engine.