Create A2A network operations dashboard with federation view

Introduction

When you federate agents across organizations, you lose the ability to SSH into a peer's runtime to debug a stuck task — you can only reason from what each peer publishes through its metrics endpoint. Teams that ship federation without a real operations dashboard learn this the hard way: a peer's expired signing cert silently rejects every inbound call for six hours before anyone notices, and the SLA breach is measured in lost contracts and broken cross-org workflows. By the end of this lesson you'll be able to design a dashboard that aggregates per-peer telemetry, surfaces topology and trust drift, and produces the leadership rollup that answers "if peer X falls over, how much of our portfolio stops working?"

Key Terminology

• A2A federation — a topology in which agent runtimes owned by different orgs call each other over signed, schema-bounded messages instead of sharing a single control plane; it matters because every observability assumption you have inside one cluster breaks at the org boundary.
• PeerSnapshot — the flat DTO returned for a single peer's /federation/metrics fetch; the dashboard reasons over a dict of these and treats an unhealthy snapshot as a partial result rather than a refresh-killing error.
• Signing failure rate — fraction of inbound messages that failed signature verification over a sliding window; under steady state it is exactly zero, so any sustained non-zero value is paged, never warned.
• Dead-letter queue — the quarantine for A2A messages that failed verification and cannot be replayed silently; depth here always requires human review, and the 15-minute slope matters more than the absolute level.
• Single-source skill — a federated capability advertised by exactly one peer; these are the federation's fragile spots and the input to the leadership risk-concentration rollup that justifies investing in a backup peer.

Concepts

Pitfalls

• Fetching peer metrics without a per-peer timeout: one slow partner stalls the whole refresh and the dashboard goes stale globally instead of degrading locally.
• Trusting peer-reported messages_per_sec as ground truth for billing or SLA: cross-check against your own egress logs at least daily — peers can under-report.
• Alerting on absolute queue depth rather than rate-of-change: a healthy federation has steady backlogs that look scary to a fresh on-call.
• Treating signing failures as a soft warning: any non-zero rate is either a config drift or an attack, never noise. Page.
• Letting cert-expiry alerts fire only inside the dashboard: route them into the same incident channel as availability pages; expired roots cause silent outages.
• Hiding asymmetric trust in the topology view: if A trusts B but B does not trust A, you have a half-broken edge and your call graphs will surprise you.
• Computing capability inventory from a static config rather than from peer-published agent cards: federations drift, configs lie, and the only truth is what the peer is currently advertising.
• Skipping the leadership rollup because it "feels like a slide": the single-peer dependency map is what justifies investing in a backup peer before, not after, the outage.

Code Walkthrough

What the dashboard has to show

A federation operator wears three hats simultaneously: SRE (latency, queue depth, dead-letters), security officer (trust, identity, suspicious patterns), and product owner (which capabilities are reachable, where workflows concentrate risk). The panels below map one-to-one onto those concerns, and every panel pulls from the same canonical aggregation layer so that an alert and an exec slide always agree on numbers.

Federation topology view

Each node is an org or cluster running an A2A endpoint. Each edge carries a direction (who initiates calls), a volume (messages/sec), and a trust polarity (one-way trust, mutual, or revoked). Bidirectional edges are drawn as two arrows so you can see asymmetric trust — a common precursor to a federation incident.

• Edge label line 1 carries throughput plus the SLO-relevant percentile, not the mean — means hide the failures.
• Edge label line 2 is reserved for the worst trust or quality signal on that edge in the last 5 minutes; this keeps the topology view useful at a glance.
• One-way trust arrows are colored differently in the rendered UI so a reviewer can spot asymmetry without reading labels.

Per-edge metrics

For every directed edge (peer_a -> peer_b) the dashboard tracks four numbers that cover the failure modes A2A actually exhibits in production:

1. messages_per_sec — base load; sudden drops often beat alerts.
2. signing_failure_rate — fraction of inbound messages that failed signature verification; non-zero is always an incident.
3. response_p95_ms — end-to-end including remote agent execution; isolates their slowness from yours.
4. schema_mismatch_count — agent card / artifact contract drift; the leading indicator of a peer's silent breaking change.

The A2ANetworkDashboard aggregator

The aggregator's job is to fan out to each peer's /federation/metrics endpoint, normalize the payloads, and expose a single in-memory snapshot the UI and alert rules read from. Peers are queried concurrently with a tight per-peer timeout so one slow partner cannot stall the whole refresh cycle.

1import asyncio
2import time
3from dataclasses import dataclass, field
4from typing import Any
5
6import httpx
7
8@dataclass
9class PeerSnapshot:
10    peer_id: str
11    org_name: str
12    fetched_at: float
13    healthy: bool
14    edges: list[dict[str, Any]] = field(default_factory=list)
15    capabilities: list[dict[str, Any]] = field(default_factory=list)
16    trust: dict[str, Any] = field(default_factory=dict)
17    queues: dict[str, int] = field(default_factory=dict)
18    error: str | None = None
19
20class A2ANetworkDashboard:
21    def __init__(self, peers: list[dict[str, str]], timeout_s: float = 2.0):
22        self._peers = peers          # [{"peer_id", "org_name", "metrics_url", "auth"}]
23        self._timeout_s = timeout_s
24        self._snapshot: dict[str, PeerSnapshot] = {}
25        self._last_refresh: float = 0.0
26
27    async def _fetch_peer(self, client: httpx.AsyncClient, peer: dict) -> PeerSnapshot:
28        started = time.time()
29        try:
30            resp = await client.get(
31                peer["metrics_url"],
32                headers={"Authorization": peer["auth"]},
33                timeout=self._timeout_s,
34            )
35            resp.raise_for_status()
36            body = resp.json()
37            return PeerSnapshot(
38                peer_id=peer["peer_id"],
39                org_name=peer["org_name"],
40                fetched_at=started,
41                healthy=True,
42                edges=body.get("edges", []),
43                capabilities=body.get("capabilities", []),
44                trust=body.get("trust", {}),
45                queues=body.get("queues", {}),
46            )
47        except Exception as exc:
48            return PeerSnapshot(
49                peer_id=peer["peer_id"],
50                org_name=peer["org_name"],
51                fetched_at=started,
52                healthy=False,
53                error=f"{type(exc).__name__}: {exc}",
54            )
55
56    async def refresh(self) -> None:
57        async with httpx.AsyncClient(http2=True) as client:
58            results = await asyncio.gather(
59                *(self._fetch_peer(client, p) for p in self._peers)
60            )
61        self._snapshot = {s.peer_id: s for s in results}
62        self._last_refresh = time.time()

Reading the snapshot: trust alerts and capability index

Once the aggregator has populated self._snapshot, the rest of the dashboard is pure read-side analysis: classify expiring certs and signing failures into page/warn alerts, then derive the cross-org capability index that reveals duplicated and single-source skills. Both consumers operate on the same PeerSnapshot dict, so a panel and an alert rule never disagree on what a peer reported.

Code snippetpython
1from collections import defaultdict
2from datetime import datetime, timezone
3
4def trust_alerts(dashboard: A2ANetworkDashboard, now: datetime | None = None) -> list[dict]:
5    now = now or datetime.now(timezone.utc)
6    alerts: list[dict] = []
7    for peer_id, snap in dashboard._snapshot.items():
8        if not snap.healthy:
9            alerts.append({"severity": "warn", "peer": peer_id, "kind": "unreachable"})
10            continue
11        for cert in snap.trust.get("certs", []):
12            days_left = (datetime.fromisoformat(cert["not_after"]) - now).days
13            if days_left < 7:
14                sev = "page"
15            elif days_left < 30:
16                sev = "warn"
17            else:
18                continue
19            alerts.append({"severity": sev, "peer": peer_id, "kind": "cert_expiry",
20                           "subject": cert.get("subject"), "days_left": days_left})
21        if snap.trust.get("signing_failure_rate_5m", 0.0) > 0.001:
22            alerts.append({"severity": "page", "peer": peer_id, "kind": "signing_failures",
23                           "rate": snap.trust["signing_failure_rate_5m"]})
24        for pattern in snap.trust.get("anomalies", []):
25            alerts.append({"severity": "warn", "peer": peer_id, "kind": "pattern", **pattern})
26    return alerts
27
28def capability_index(dashboard: A2ANetworkDashboard) -> dict[str, list[str]]:
29    skill_to_peers: dict[str, list[str]] = defaultdict(list)
30    for snap in dashboard._snapshot.values():
31        for card in snap.capabilities:
32            for skill in card.get("skills", []):
33                skill_to_peers[skill["id"]].append(f"{snap.org_name}:{card['agent_id']}")
34    return dict(sorted(skill_to_peers.items()))
35
36def single_source_skills(dashboard: A2ANetworkDashboard) -> list[str]:
37    return [s for s, peers in capability_index(dashboard).items() if len(peers) == 1]

• trust_alerts classifies cert expiry into page (<7d) vs warn (7–30d) bands and treats any non-zero sustained signing_failure_rate_5m as an immediate page, since the steady-state value is exactly 0.
• capability_index builds the canonical skill-to-peer fan-out map; org_name:agent_id keeps the result legible to non-engineers reading a dashboard.
• single_source_skills calls out skills offered by exactly one peer — these are the federation's fragile spots and feed directly into the leadership rollup.

Operations panels: queues, dead-letter, SLA

A2A messages that fail verification cannot be replayed silently — they go to a quarantine dead-letter queue that an operator must clear. The ops panel shows replay-queue depth (delayed-but-recoverable), dead-letter depth (unverifiable, requires human review), and a per-peer SLA-breach counter against the contractually agreed P95 and availability targets. Depth alone is not enough; the slope matters more than the level, so a 15-minute delta is rendered alongside the absolute number.

Leadership rollup: dependency map and risk concentration

The exec view answers one question: which peer, if it goes down for an hour, breaks the most internal workflows? The dashboard joins workflow-definition metadata (which workflow steps call which federated skill) against the capability index to compute, for every peer, the count and revenue-tagged value of workflows that hard-depend on them. Peers in the top quartile are flagged as "risk-concentrated" and become input to procurement and partnership reviews — not just SRE rotations.

Do's and Don'ts

Do's

1. ✓Do surface response_p95_ms — not the mean — on every topology edge label — Means hide the tail failures that breach cross-org SLAs; the lesson's topology design reserves edge label line 1 for the P95 percentile precisely because a peer can look healthy on average while its slowest 5 % of calls are timing out your downstream workflows.
2. ✓Do catch all exceptions inside _fetch_peer and return an unhealthy PeerSnapshot rather than re-raising — This is what lets asyncio.gather in refresh() complete for every peer concurrently; without the per-peer isolation, a single partner that times out or returns 5xx propagates an exception that cancels the entire refresh cycle and leaves the dashboard stale.
3. ✓Do track schema_mismatch_count as a dedicated per-edge metric alongside signing_failure_rate — schema_mismatch_count is the leading indicator of a peer's silent breaking change to their agent card or artifact contract, while signing_failure_rate is the identity signal where any non-zero value is an active incident; they map to different failure modes and neither can substitute for the other.

Don'ts

1. ✗Don't collapse bidirectional federation edges into a single undirected arrow — The lesson draws two directed arrows for mutual-trust edges specifically to expose asymmetric trust (different volumes, different signing_failure_rate per direction), which it names as "a common precursor to a federation incident"; a merged arrow hides the asymmetry a security review needs to catch.
2. ✗Don't create a separate httpx.AsyncClient per peer inside the concurrent fan-out — The lesson's refresh() opens one shared AsyncClient for the entire asyncio.gather call so HTTP/2 connection reuse holds across all peers simultaneously; instantiating per-peer clients forfeits that pooling and multiplies TLS handshake overhead when dozens of org endpoints are queried on every refresh cycle.
3. ✗Don't let trust alert rules and the exec rollup fetch peer data independently of the UI panels — The lesson routes trust_alerts() and the capability index through the same self._snapshot dict; if an alert rule and an exec slide pull from separate fan-outs, they can report different numbers for the same peer's signing_failure_rate or trust expiry window in the same five-minute window.