Create eval architecture audit report with coverage analysis

Introduction

When you ship a new generator, judge, or tool into a production GenAI stack, the worst failure is not a crashing bug — it is a silent, un-monitored deployment that ships without an attached eval and goes unnoticed until users hit a regression. Teams that track "what is under eval" in wiki pages and memory inevitably drift to a state where a third of their generation paths carry no correctness, faithfulness, or safety signal at all. By the end of this lesson you'll be able to build an EvalCoverageAuditor that walks the live architecture graph, joins each component against the eval-run ledger, and emits a structured coverage matrix that classifies every cell as covered, partial, or uncovered — plus a single rollup score for leadership and a CI gate that blocks PRs which add a generation stage without an attached eval.

Key Terminology

• Coverage matrix: the 2D grid of components × eval dimensions whose cells (covered / partial / uncovered / n/a) are the contract the auditor versions, emits, and CI checks against.
• Stale binding: an eval binding whose last_run_at is older than STALE_AFTER (30 days); the cell is demoted to partial because monitoring exists on paper but has not actually run recently.
• Drift-prone binding: an eval binding whose golden_set_refreshed_at is older than DRIFT_AFTER (90 days); even fresh runs against an aged golden set no longer reflect production traffic, so the cell is downgraded to partial.
• Coverage score: covered_cells / applicable_cells — the single rolled-up percentage that leadership tracks weekly and that CI blocks regressions against.
• Applicable dimension: a dimension declared relevant for a component's kind (e.g. retrievers usually omit safety); cells outside this set report n/a and are excluded from the score denominator so the metric cannot be gamed.

Concepts

The coverage matrix

The audit is structured as a two-dimensional matrix. Rows are the system components that produce or transform model output: retrievers, generators, tools, judges, and gates. Columns are the eval dimensions that matter for production safety: correctness, faithfulness, latency, cost, and safety. Every cell is one of three states — covered, partial, or uncovered — and carries the timestamp of the most recent eval run that produced it.

The matrix is the contract. It is what gets versioned in git, what the auditor emits, what CI checks, and what shows up in the leadership rollup. Everything else in this section exists to populate it correctly and act on what it says.

CI integration

The audit runs on every PR. Two checks fail the build: introducing a new generation-kind component without an eval binding, and dropping the platform coverage score below the previous main-branch baseline. The first check is what stops the slow drift to the worst-case state: a system that has grown three new generators since anyone last looked, none of them under eval, all of them shipping daily. Compare the PR's architecture_graph.yaml against main; for each new component of kind generator, require at least one binding for correctness and safety in bindings.yaml. No binding, no merge.

The second check — score-must-not-regress — uses the JSON payload to compare PR-branch score against the cached main-branch score. A PR that adds a covered component raises the score; a PR that adds an uncovered one lowers it; the latter fails. Engineers get an actionable error with the exact component / dimension cells that are missing, not a generic "coverage dropped".

The leadership rollup

A single number, computed the same way every audit run: coverage_score = covered_cells / applicable_cells. Ship it weekly. Plot it. The first time it reaches 95% your eval architecture is real; the first time it falls below 85% without a corresponding architectural change is the signal that something has been quietly removed from eval. The number is intentionally crude — weighted scores and per-tier rollups invite the kind of accounting games that make the metric meaningless. Resist them in v1.

Operating discipline

• Run the auditor on a schedule (daily) and on every PR. Daily catches expiring freshness; per-PR catches new uncovered components before they ship.
• Treat uncovered cells as P2 incidents, not backlog tickets. An unmonitored generator is shipping un-evaluated tokens to users — that is an outage in waiting.
• Refresh golden sets quarterly at minimum; sample 200-500 recent production prompts, label them, and rotate them into the golden set so golden_set_refreshed_at advances.
• Never let a component declare applicable_dimensions = () to game the score. PR review on the architecture-graph file is mandatory and a code owner must sign off.
• The EvalCoverageAuditor is read-only by design. Do not let it run evals or mutate state — its trustworthiness depends on being a pure inspector of the ledger.
• Keep one source of truth for the architecture graph. If components.yaml and the runtime DI container disagree, the audit lies.
• Publish the markdown report to a known URL per audit run and link it from the platform README. An audit nobody reads is an audit that does not exist.
• When a component is genuinely retired, delete it from the graph in the same PR. Lingering retired components inflate the denominator and hide real coverage gaps.

Code Walkthrough

The coverage matrix above is only a contract until something populates it — this walkthrough builds the EvalCoverageAuditor that walks the architecture graph, joins each component against the eval-run ledger, and fills every cell. Start with the closed vocabulary and the two freshness knobs ops will tune over time:

Code snippetpython
1from dataclasses import dataclass
2from datetime import datetime, timedelta, timezone
3from enum import Enum
4
5class Coverage(str, Enum):
6    COVERED = "covered"
7    PARTIAL = "partial"
8    UNCOVERED = "uncovered"
9    NOT_APPLICABLE = "n/a"
10
11DIMENSIONS = ("correctness", "faithfulness", "latency", "cost", "safety")
12STALE_AFTER = timedelta(days=30)   # demote to partial if not run recently
13DRIFT_AFTER = timedelta(days=90)   # demote if golden set has aged out
14
15@dataclass(frozen=True)
16class Component:
17    id: str
18    kind: str                      # retriever | generator | tool | judge | gate
19    applicable_dimensions: tuple[str, ...]
20
21@dataclass(frozen=True)
22class EvalBinding:
23    component_id: str
24    dimension: str
25    last_run_at: datetime | None
26    golden_set_refreshed_at: datetime | None

The auditor stays pure: it does not run evals, it only inspects what has run. Applicability is checked first so a judge is never penalised for a faithfulness cell it does not need, a missing binding short-circuits to uncovered (the alarm), and stale always wins over drift.

Code snippetpython
1class EvalCoverageAuditor:
2    def __init__(self, components, bindings, now=None):
3        self._components = list(components)
4        self._bindings = {(b.component_id, b.dimension): b for b in bindings}
5        self._now = now or datetime.now(timezone.utc)
6
7    def _classify(self, c, dim):
8        if dim not in c.applicable_dimensions:
9            return Coverage.NOT_APPLICABLE
10        b = self._bindings.get((c.id, dim))
11        if b is None or b.last_run_at is None:
12            return Coverage.UNCOVERED
13        if self._now - b.last_run_at > STALE_AFTER:
14            return Coverage.PARTIAL
15        if (b.golden_set_refreshed_at is None
16                or self._now - b.golden_set_refreshed_at > DRIFT_AFTER):
17            return Coverage.PARTIAL
18        return Coverage.COVERED
19
20    def audit(self):
21        return {(c.id, d): self._classify(c, d)
22                for c in self._components for d in DIMENSIONS}
23
24    def coverage_score(self, cells):
25        applicable = [s for s in cells.values() if s is not Coverage.NOT_APPLICABLE]
26        full = sum(1 for s in applicable if s is Coverage.COVERED)
27        return round(full / len(applicable), 4) if applicable else 1.0

The audit() dict is the machine-readable matrix CI gates on; coverage_score is the single percentage the leadership rollup tracks, with n/a cells excluded from the denominator so the metric cannot be gamed. You'll know it works when the auditor emits a covered, partial, or uncovered state for every applicable component × dimension pair and coverage_score returns the same rolled-up percentage that blocks a regressing PR.

Do's and Don'ts

Do's

1. ✓Do check applicable_dimensions before probing _bindings in _classify() — returning Coverage.NOT_APPLICABLE first ensures a judge component is never penalised for a faithfulness or latency cell it was never designed to cover, and keeps those cells excluded from the coverage_score denominator so the rollup percentage cannot be gamed by over-declaring N/A.
2. ✓Do record both last_run_at and golden_set_refreshed_at on every EvalBinding — staleness (>30 days since last run, STALE_AFTER) and golden-set drift (>90 days since refresh, DRIFT_AFTER) are independent failure modes; _classify demotes to Coverage.PARTIAL on either condition, so a binding with a recent last_run_at but a None golden_set_refreshed_at still returns PARTIAL, not COVERED.
3. ✓Do inject a fixed now timestamp through the EvalCoverageAuditor.__init__ now= parameter in every test — the staleness and drift thresholds are computed relative to self._now, so pinning that value to a known datetime makes STALE_AFTER and DRIFT_AFTER boundary assertions deterministic and prevents tests from silently passing only on the day they were written.

Don'ts

1. ✗Don't declare applicable_dimensions = () on a Component to shrink the coverage matrix — every dimension cell collapses to Coverage.NOT_APPLICABLE and is excluded from the coverage_score denominator, inflating the rollup while leaving real generation paths — retrievers, generators, and tools — with zero eval signal.
2. ✗Don't let EvalCoverageAuditor run evals or write to _bindings — the auditor is a pure reader of the eval-run ledger, and its audit() output is only trustworthy as a CI gate signal when the inspector and the systems it monitors are fully separated; mixing mutation into _classify or audit means the report can reflect a state the auditor itself produced.
3. ✗Don't treat a Coverage.PARTIAL cell as passing in CI gates — PARTIAL signals that either the eval binding's last_run_at has exceeded STALE_AFTER or golden_set_refreshed_at has exceeded DRIFT_AFTER; routing around it suppresses the warning that a binding exists but can no longer be trusted to catch regressions before they reach users.