Audit MCP security with automated testing

Introduction

In production, a security control that passes code review at deploy time can silently break weeks later — and without continuous verification, nobody notices until an incident. MCP tool servers face exactly this exposure: tool poisoning payloads mutate, auth boundaries drift, and cross-server exfiltration paths open as configurations change. Engineers often discover these regressions through postmortems rather than proactive checks. By the end of this lesson, you'll be able to implement an automated audit suite that runs hourly against every registered MCP tool server, tracks posture signals on a dashboard, and pages on-call when critical findings emerge.

Key Terminology

• Audit suite — A family of automated security tests targeting one threat category (tool poisoning, auth boundary, A2A integrity, or cross-server exfiltration). Each suite is applied by AuditRunner to every registered tool server and produces per-server findings labeled by severity.
• Synthetic agent — A dedicated SyntheticAgent instance that drives end-to-end audit tasks through the live MCP gateway under a fixed, isolated identity, executing multi-step tasks and recording a RunRecord for each without touching real customer data.
• expect_block flag — A per-step marker on audit tasks indicating that the input should be denied by a gateway defense. When expect_block=True but the response is not blocked, SyntheticAgent records a defense_did_not_fire finding — distinguishing a missing defense from a correctly allowed request.
• audit_findings_total counter — A Prometheus Counter labeled by suite, server, and severity that AuditRunner increments for every finding it discovers. Critical-severity increments also immediately fire an alert; all severities accumulate in a long-term store for trend analysis.
• Service registry — The MCP gateway's live catalog of registered tool servers, queried at the start of every AuditRunner.run() call so that audit coverage automatically extends to newly deployed servers without manual list maintenance.
• Posture dashboard — A set of four Prometheus-backed signals — audit pass rate, critical findings open, MTTR, and server coverage — that give on-call engineers a continuous view of security health across all registered MCP tool servers.

Concepts

Security Controls Degrade Between Deploys

A control that passes code review at deploy time has an unknown shelf life. MCP tool server configurations drift: tool descriptions are updated, auth token scopes change, network policies shift, and new exfiltration paths open as adjacent services evolve. Attackers also refine poisoning payloads faster than quarterly penetration tests can keep up. In practice, regressions surface through postmortems rather than proactive checks — which means a team relying on point-in-time review is always reacting rather than preventing.

Automated, continuous auditing flips the question from "was this control correct at deploy time?" to "is this control working right now?" Running the audit suite hourly means any regression surfaces within a single on-call shift rather than aging silently until an incident.

From Registry to Alert: How the Pipeline Assembles

The audit runner is a Kubernetes CronJob that begins each run by querying the MCP gateway's service registry — the live catalog of registered tool servers. Querying the registry dynamically, rather than maintaining a static server list, means coverage automatically includes newly deployed servers. For each server, AuditRunner applies all four suite families; each suite emits findings labeled by suite, server, and severity via the audit_findings_total Prometheus counter (see Code Walkthrough).

Critical findings short-circuit to an immediate alert. Lower-severity findings accumulate in a long-term store, enabling trend queries across weeks and months — useful for detecting a server whose auth boundary degrades gradually rather than failing all at once.

The Synthetic Agent and the expect_block Pattern

SyntheticAgent operates like a continuous red-teamer, driving multi-step sessions through the live MCP gateway. The key design decision is the expect_block flag: rather than verifying that legitimate requests succeed, each step in an audit task can be marked to assert that a defense fires. If a step with expect_block=True receives an unblocked response, the runner records defense_did_not_fire — a precise signal identifying which control is missing and on which server (see Code Walkthrough).

The synthetic agent runs under a dedicated identity so its traffic is filterable in operational dashboards. This isolation ensures that a misbehaving audit run cannot corrupt user-behavior metrics or confuse incident triage — audit findings and customer events stay separable in every downstream query.

The Four Posture Signals

The dashboard tracks four signals, each measuring a distinct dimension of security health. Audit pass rate (target ≥ 99 %) measures regression: a drop means a previously working defense has stopped firing. Critical findings open (target: zero) measures unresolved risk: any non-zero value means a known critical flaw exists in production right now. MTTR (target ≤ 24 h) measures response discipline: without a remediation SLA, the dashboard becomes wallpaper and findings accumulate indefinitely. Coverage (target: 100 %) guards against blind spots — a server registered in the gateway but unreachable by the audit runner is invisible to the pass rate, so 100 % pass rate at 90 % coverage is a false signal. Coverage must count successful audits, not registered targets.

Code Walkthrough

Now that you understand the four audit suite families and the posture-dashboard signals they feed, here is how those pieces assemble into a running system.

The audit runner is a Kubernetes CronJob that queries the MCP gateway's service registry, applies each suite to every registered tool server, and emits a Prometheus counter labeled by suite, server, and severity. Critical findings fire an alert immediately; lower-severity findings accumulate in a long-term store for trend queries across months.

Code snippetpython
1from prometheus_client import Counter
2
3audit_findings_total = Counter(
4    "mcp_audit_findings_total",
5    "Findings emitted by the audit runner",
6    ["suite", "server", "severity"],
7)
8
9class AuditRunner:
10    def run(self):
11        servers = self.registry.list_servers()
12        findings = []
13        for server in servers:
14            for suite in self.suites:
15                findings.extend(suite.run_against(server))
16        for f in findings:
17            audit_findings_total.labels(
18                suite=f.suite, server=f.server, severity=f.severity
19            ).inc()
20            if f.severity == "critical":
21                self.alerts.fire(f)
22        self.report.append(findings)

The synthetic agent drives end-to-end tests through the live MCP gateway. Each audit task is a sequence of steps; every step expected to trigger a defense is marked expect_block=True. If the defense does not fire when it should, the runner records the step as a finding. The agent operates under a dedicated identity so its traffic is filterable in operational dashboards and never touches real customer data.

Code snippetpython
1from dataclasses import dataclass
2
3@dataclass
4class RunRecord:
5    passed: bool
6    reason: str = ""
7
8    @staticmethod
9    def fail(reason: str, step) -> "RunRecord":
10        return RunRecord(passed=False, reason=reason)
11
12    @staticmethod
13    def pass_() -> "RunRecord":
14        return RunRecord(passed=True)
15
16class SyntheticAgent:
17    def run_audit_task(self, task) -> RunRecord:
18        session = self.start_session(task.identity)
19        for step in task.steps:
20            response = session.send(step.user_input)
21            if step.expect_block and response.blocked:
22                continue
23            if step.expect_block and not response.blocked:
24                return RunRecord.fail(reason="defense_did_not_fire", step=step)
25        return RunRecord.pass_()

Together, AuditRunner and SyntheticAgent feed all four posture-dashboard signals: the Prometheus counter backs the audit pass rate and critical-findings-open metrics, synthetic task results confirm that each defense is still wired end-to-end, and hourly scheduling ensures any regression surfaces within a single on-call shift rather than waiting until the next deploy review.

Confirm that the CronJob completes without critical findings, the audit_findings_total counter increments on each run, and the dashboard holds its four targets: audit pass rate ≥ 99 %, zero critical findings open, MTTR ≤ 24 h, and coverage at 100 %.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do label audit_findings_total with suite, server, and severity — these three dimensions let the dashboard isolate which tool server regressed, which audit suite caught it, and whether the finding crosses the critical threshold that triggers an immediate alert versus accumulating in the long-term trend store.
2. ✓Do mark each synthetic-agent step with expect_block=True when it should trigger a defense — the SyntheticAgent only records a finding when a block was expected but did not fire, so omitting the flag silently lets bypassed defenses pass as green results on the audit pass-rate metric.
3. ✓Do run the AuditRunner as a Kubernetes CronJob on an hourly schedule — MCP tool-poisoning payloads, auth-boundary drift, and cross-server exfiltration paths can open between deploys; an hourly cadence guarantees any regression surfaces within one on-call shift rather than at the next postmortem.

Don'ts

1. ✗Don't let the synthetic agent share an identity with real user traffic — the SyntheticAgent must operate under a dedicated identity so its audit traffic is filterable in dashboards and never contaminates customer-data paths or skews behavioral signals used by other defenses.
2. ✗Don't fire alerts only for critical findings without accumulating lower-severity findings in a long-term store — critical findings page on-call immediately, but non-critical findings that accumulate over time reveal drift in coverage and MTTR; discarding them silently inflates the audit pass-rate metric while posture erodes.
3. ✗Don't treat the four posture-dashboard targets (audit pass rate ≥ 99 %, zero critical findings open, MTTR ≤ 24 h, coverage 100 %) as independent checks — they are all backed by the same audit_findings_total counter and synthetic task results; a gap in the AuditRunner's server registry query means servers added after the last deploy are never audited, causing coverage to silently drop below 100 % while the other metrics appear healthy.