Create MCP ecosystem governance dashboard

Introduction

When your MCP tool mesh grows from a handful of tools to dozens across multiple teams, leadership stops asking "does it work?" and starts asking "what do we own, who runs it, and what is rotting?" Without a governance dashboard, schema drift goes unnoticed until callers crash in production, orphan tools accumulate review debt, and elevated-permission surfaces silently widen. By the end of this lesson you'll be able to design the four panel groups (inventory, quality, adoption, risk), aggregate registry and telemetry state on a schedule, and wire the dashboard into a deprecation workflow that turns signals into action.

Key Terminology

• MCP registry: the declarative store of every published tool — tool_id, owner team, semver, sandbox profile, elevated permissions, and the schema hash the version was published with.
• Telemetry projection: the observed-state view of each tool over a rolling window — call volume, success rate, P95 latency, error-class breakdown, observed schema hash, and the set of calling agents.
• Schema drift: the condition where a tool's published_schema_hash in the registry differs from the observed_schema_hash seen on the wire; the canonical risk-panel trigger.
• Orphan tool: a registered tool whose caller set is empty for longer than ORPHAN_GRACE_DAYS; the canonical signal that fans into the deprecation workflow.
• Elevated permission: a sandbox capability outside the default profile (network egress, filesystem write, secret read); tracked as a closed enum so risk-panel counts aren't fractured by free-form strings.
• Materialized snapshot: the cached aggregate (DashboardSnapshot) refreshed on a 5–15 minute schedule so panels render from one consistent join instead of live registry + telemetry queries per page load.

Concepts

What the dashboard must show

The dashboard's job is to compress every signal a platform owner cares about into four panel groups: ecosystem inventory, tool-quality metrics, adoption analytics, and risk panels. Each panel pulls from at least two sources — the registry (declarative state) and the telemetry pipeline (observed state) — and the value of governance comes from the join, not from either side alone.

Panel taxonomy

• Ecosystem inventory — every registered tool, owner team, semver, last health check, sandbox profile, and current request rate.
• Tool quality — success rate, P95 latency, error-class breakdown (timeout / 4xx / 5xx / schema-violation), and schema-violation rate over the trailing 7 days.
• Adoption analytics — caller-to-tool matrix, top tools by call volume, orphaned tools (zero callers in N days), and version-skew across consumers.
• Risk panels — stale sandbox profiles, tools holding elevated permissions (network egress, filesystem write, secret reads), and schema drift between the version published in the registry and the version observed on the wire.

Pitfalls

• Joining on stale registry snapshots. If your registry cache is older than the telemetry window, schema drift will fire false positives every time a tool publishes a new version. Pin the registry read to a transaction id or always read fresher than the telemetry window.
• Treating "low call volume" as "orphan." A nightly batch tool might fire eight times a month and still be load-bearing. Orphan detection must be callers == empty, never calls < threshold, or you will retire critical tooling.
• Letting elevated permissions live in free-form strings. If elevated_permissions is ["net", "network", "egress"] across three teams, your risk panel undercounts. Enforce a closed enum at registration time.
• Schema-drift alarms without a quarantine path. Detecting drift is easy; deciding what to do is hard. Wire the risk panel to an automatic version-pin rollback or a routing freeze, not just a Slack ping.
• EOL announcements without a forced migration deadline. "Deprecated" with no date is noise. Every EOL announcement must carry a removal_at timestamp and the dashboard must show countdown days, not vague status text.
• Sandbox profiles aging silently. sandbox_profile_age_days > 90 is a config smell, not a security incident — but ignored long enough it becomes one. Make stale-sandbox a blocker on the next version publish, not just a panel row.
• Conflating tool owners with on-call. The owner team in the registry is for governance correspondence; paging during an incident needs a separate on_call_rotation field. Don't make the dashboard the source of truth for both.
• Rendering the dashboard from live queries. A governance dashboard that costs a registry scan and a telemetry rollup per page load will be turned off the first time someone refreshes it during an incident. Materialize the snapshot every 5-15 minutes and serve from cache.

Code Walkthrough

Now that you have the panel taxonomy in hand, the walkthrough turns it into working code. The governance value comes from the join: the registry carries declarative state (owner, version, published schema hash, sandbox age, elevated permissions) while telemetry carries observed state (call volume, callers, observed schema hash). We model each side as a frozen dataclass keyed on tool_id, then aggregate them into a single DashboardSnapshot so panels render from one consistent view.

Code snippetpython
1from dataclasses import dataclass
2from datetime import datetime, timedelta, timezone
3
4@dataclass(frozen=True)
5class ToolRecord:
6    tool_id: str
7    owner_team: str
8    version: str
9    sandbox_profile_age_days: int
10    elevated_permissions: tuple[str, ...]
11    published_schema_hash: str
12
13@dataclass(frozen=True)
14class ToolTelemetry:
15    tool_id: str
16    calls_24h: int
17    observed_schema_hash: str
18    callers: frozenset[str]
19    last_called_at: datetime | None
20
21@dataclass(frozen=True)
22class DashboardSnapshot:
23    risk: list[dict]
24    adoption: dict
25    generated_at: datetime
26
27def _zero_telemetry(tool_id: str) -> ToolTelemetry:
28    return ToolTelemetry(tool_id, 0, "", frozenset(), None)

The aggregator joins the two streams on tool_id, then projects panel views. The risk panel computes schema drift as published != observed (guarded so a tool with no telemetry never false-positives), flags stale sandboxes, and surfaces elevated permissions. The adoption panel detects orphans — zero callers past ORPHAN_GRACE_DAYS — which later fan into the deprecation workflow.

Code snippetpython
1class MCPGovernanceDashboard:
2    ORPHAN_GRACE_DAYS = 14
3    SANDBOX_STALE_DAYS = 90
4
5    def __init__(self, registry, telemetry, clock=None):
6        self._registry = registry
7        self._telemetry = telemetry
8        self._now = clock or (lambda: datetime.now(timezone.utc))
9
10    def snapshot(self) -> DashboardSnapshot:
11        records = {r.tool_id: r for r in self._registry.list_tools()}
12        telemetry = {t.tool_id: t for t in self._telemetry.window(hours=24)}
13        joined = [
14            (rec, telemetry.get(tid, _zero_telemetry(tid)))
15            for tid, rec in records.items()
16        ]
17        return DashboardSnapshot(
18            risk=self._risk(joined),
19            adoption=self._adoption(joined),
20            generated_at=self._now(),
21        )
22
23    def _risk(self, joined) -> list[dict]:
24        rows = []
25        for rec, tel in joined:
26            flags = []
27            if rec.sandbox_profile_age_days > self.SANDBOX_STALE_DAYS:
28                flags.append("stale_sandbox")
29            if rec.elevated_permissions:
30                flags.append("elevated_permissions")
31            if tel.observed_schema_hash and rec.published_schema_hash != tel.observed_schema_hash:
32                flags.append("schema_drift")
33            if flags:
34                rows.append({"tool_id": rec.tool_id, "owner": rec.owner_team, "flags": flags})
35        return rows
36
37    def _adoption(self, joined) -> dict:
38        cutoff = self._now() - timedelta(days=self.ORPHAN_GRACE_DAYS)
39        orphans = [
40            rec.tool_id
41            for rec, tel in joined
42            if not tel.callers and (tel.last_called_at is None or tel.last_called_at < cutoff)
43        ]
44        top = sorted(joined, key=lambda jt: jt[1].calls_24h, reverse=True)[:20]
45        return {"orphans": orphans, "top_by_volume": [r.tool_id for r, _ in top]}

You'll know it works when calling snapshot() against a registry holding one tool whose observed hash differs from its published hash and one tool with no callers past the grace window returns exactly one schema_drift entry under risk and that second tool_id in adoption["orphans"].

Do's and Don'ts

Do's

1. ✓Do join _registry.list_tools() and _telemetry.window(hours=24) into a single DashboardSnapshot before projecting any panel — if _risk() and _adoption() each query the registry and telemetry independently, a schema push or new registration that arrives between calls creates cross-panel inconsistencies where the same tool appears in the risk panel but not the adoption panel within the same render cycle.
2. ✓Do fall back to _zero_telemetry() for tools absent from the telemetry window, then guard the drift check with if tel.observed_schema_hash before comparing against published_schema_hash — without both, every newly registered tool whose first call hasn't landed yet carries an empty observed hash that spuriously triggers schema_drift in the risk panel before any real drift has occurred.
3. ✓Do define orphan status with the compound condition not tel.callers and (last_called_at is None or last_called_at < cutoff) using ORPHAN_GRACE_DAYS as the cutoff, not a call-volume floor — a nightly batch tool can show calls_24h == 0 in any given 24-hour window yet have a last_called_at inside the 14-day grace period; a volume threshold misclassifies it as orphaned and fans it into the deprecation workflow incorrectly.

Don'ts

1. ✗Don't omit the if tel.observed_schema_hash guard before the published_schema_hash != observed_schema_hash comparison — _zero_telemetry() initializes observed_schema_hash as an empty string, so an unguarded != fires schema_drift for every tool that has never been called in the window, drowning real drift signals in false positives and making the risk panel unactionable.
2. ✗Don't collapse elevated_permissions, stale_sandbox, and schema_drift into a single aggregated risk score without preserving the per-flag list — each flag in _risk() maps to a distinct remediation path (schema_drift triggers a routing freeze or version-pin rollback, stale_sandbox triggers a reconfiguration gate, elevated_permissions triggers a permission-scope audit), and a scalar score discards exactly the routing information that determines which team acts and how.
3. ✗Don't use calls_24h from the telemetry window as a proxy for the orphan check — the _adoption() method compares last_called_at against the ORPHAN_GRACE_DAYS (14-day) cutoff, not the 24-hour telemetry window; a tool on a weekly batch schedule will show calls_24h == 0 on most days but is not an orphan, and substituting a volume threshold enqueues load-bearing tools for deprecation.