Deploy MCP tool servers as sidecars with external-secrets-operator

Introduction

When you deploy a GenAI application that calls external tools, the MCP tool server needs credentials it must never bake into its image, and the FastAPI app needs that tool server reachable without crossing network boundaries or adding latency. Teams that wire this up ad hoc end up with secrets sprawled across manifests and tool servers that silently break the moment a credential rotates—pages fire, agents start returning hallucinated empty results, and the on-call engineer spends an hour matching Secret versions to pod restart times. By the end of this lesson, you'll be able to deploy an MCP tool server as a Kubernetes sidecar alongside a FastAPI pod, wire its credentials through external-secrets-operator to GCP Secret Manager, and survive secret rotation without dropping requests.

Key Terminology

• MCP sidecar: a tool-server container co-located with the FastAPI application container in the same Kubernetes pod, sharing a network namespace so the app reaches the tool server on localhost instead of crossing the cluster network.
• external-secrets-operator (ESO): a Kubernetes controller that watches ExternalSecret custom resources and reconciles their referenced upstream secrets (e.g. GCP Secret Manager) into native Kubernetes Secret objects on a defined refresh interval.
• SecretStore: an ESO custom resource that declares where secrets come from and how ESO authenticates to that provider — for the GCP backend, it pins the project ID and the Workload-Identity-bound service account ESO uses to call Secret Manager.

Concepts

Three ideas drive this lesson: (1) the sidecar topology that keeps MCP tool calls on localhost rather than crossing the cluster network, (2) the ESO → Workload Identity → projected-volume pipeline that delivers rotated credentials to the sidecar without baking them into the image, and (3) the readiness ordering required so the FastAPI container does not accept traffic before the MCP sidecar can serve tool invocations.

Operational Considerations

Running an MCP sidecar in production is not just about getting the manifests to apply cleanly — the two operational properties that determine whether the deployment survives a rotation or a rolling update are how new secret material reaches the running sidecar and how startup ordering between the FastAPI container and the sidecar is guaranteed before traffic arrives. The two subsections below cover each in turn.

Secret Rotation Without Downtime

When ESO detects a changed secret value during its refresh interval, it updates the Kubernetes Secret. Containers using envFrom require a pod restart to pick up new environment variables—Kubernetes does not hot-reload environment variables. Two strategies mitigate this:

1. Volume-based consumption: Mount the Secret as a file volume instead of envFrom. Kubernetes automatically updates mounted Secret files (with a propagation delay of up to the kubelet sync period, typically 60 seconds). The MCP server reads credentials from disk on each connection pool refresh rather than caching them from environment variables at startup.

2. Stakater Reloader: Deploy the Reloader controller, which watches for Secret changes and triggers rolling restarts of Deployments that reference the changed Secret. This works with envFrom and requires no application code changes, but introduces brief unavailability during the rollout—mitigated by the pod disruption budgets configured in another goal.

Health Checks and Startup Ordering

The MCP sidecar must be ready before the FastAPI application begins processing requests that require tool invocation. Kubernetes does not guarantee container startup order within a pod. Use these strategies:

1. Set the MCP sidecar's readinessProbe with a shorter initialDelaySeconds (5 seconds) than the application container's probe (10 seconds), giving the sidecar a head start
2. Implement a startup check in the FastAPI application that polls localhost:3001/health in a retry loop before marking itself ready—this is more reliable than relying on timing

Code Walkthrough

Architecture: MCP Sidecar with External Secrets

The following diagram illustrates how the FastAPI application container communicates with the MCP tool server sidecar over localhost, while the external-secrets-operator synchronizes credentials from GCP Secret Manager into a Kubernetes Secret mounted into the sidecar.

GKE clusters require a secure secret pipeline to feed API keys and database credentials into MCP server sidecars without baking them into container images. This architecture uses external-secrets-operator (ESO) to pull secrets from GCP Secret Manager, authenticate via Workload Identity (mcp-sa@project.iam), and reconcile them into a Kubernetes SecretStore and ExternalSecret with a one-hour sync policy. The resulting K8s Secret (mcp-tool-secrets) is volume-mounted directly into the MCP Server Sidecar container running alongside FastAPI on port 3001.

Loading diagram...

The separation of concerns is critical: the FastAPI container knows nothing about secret storage backends. The MCP sidecar reads credentials from a mounted volume that ESO keeps synchronized. When a secret rotates in GCP Secret Manager, ESO updates the Kubernetes Secret, and a volume reload delivers the new value to the sidecar—no pod restart required for projected volumes.

Building the MCP Tool Server

The MCP server exposes tools that the FastAPI application invokes during LLM agent execution. The following implementation defines MCPToolServer using the mcp Python SDK's Server class and the @server.tool() decorator to register a database query tool. The server reads credentials from environment variables injected by Kubernetes, establishes a connection pool using asyncpg, and exposes the tool over the stdio transport via mcp.server.stdio.run_server. The query_database function validates incoming SQL against an allow-list of statement prefixes to prevent write operations, returning results as structured JSON that the LLM can interpret. This pattern keeps the tool server stateless except for the connection pool, making it horizontally scalable within the sidecar pattern.

1# mcp_server.py — MCP tool server for database queries
2import os
3import json
4import asyncio
5import asyncpg
6from mcp.server import Server
7from mcp.server.stdio import run_server
8from mcp.types import Tool, TextContent
9
10ALLOWED_PREFIXES = ("SELECT", "WITH", "EXPLAIN")
11
12server = Server("db-query-tool")
13pool: asyncpg.Pool | None = None
14
15async def get_pool() -> asyncpg.Pool:
16    global pool
17    if pool is None:
18        pool = await asyncpg.create_pool(
19            host=os.environ["DB_HOST"],
20            port=int(os.environ.get("DB_PORT", "5432")),
21            user=os.environ["DB_USER"],
22            password=os.environ["DB_PASSWORD"],
23            database=os.environ["DB_NAME"],
24            min_size=2,
25            max_size=10,
26        )
27    return pool
28
29@server.tool()
30async def query_database(sql: str, params: list | None = None) -> list[TextContent]:
31    """Execute a read-only SQL query and return results as JSON."""
32    normalized = sql.strip().upper()
33    if not any(normalized.startswith(p) for p in ALLOWED_PREFIXES):
34        raise ValueError(
35            f"Only read queries allowed. Got: {normalized[:20]}"
36        )
37
38    db_pool = await get_pool()
39    async with db_pool.acquire() as conn:
40        async with conn.transaction(readonly=True):
41            rows = await conn.fetch(sql, *(params or []))
42            result = [dict(row) for row in rows]
43
44    return [TextContent(type="text", text=json.dumps(result, default=str))]
45
46@server.tool()
47async def list_tables() -> list[TextContent]:
48    """List all user tables in the public schema."""
49    db_pool = await get_pool()
50    async with db_pool.acquire() as conn:
51        rows = await conn.fetch(
52            "SELECT table_name FROM information_schema.tables "
53            "WHERE table_schema = 'public' ORDER BY table_name"
54        )
55        tables = [row["table_name"] for row in rows]
56
57    return [TextContent(type="text", text=json.dumps(tables))]
58
59if __name__ == "__main__":
60    asyncio.run(run_server(server))

Kubernetes Manifests: Sidecar Pod + ExternalSecret

The pod specification co-locates the FastAPI application and the MCP server in a single pod, sharing a network namespace so the application reaches the MCP server on localhost. The following Python script builds both halves of the deployment in one place: a V1Deployment with two containers (the primary genai-app running FastAPI on port 8000 and the mcp-db-tool sidecar running on port 3001), plus the ESO SecretStore and ExternalSecret resources that synchronize credentials from GCP Secret Manager into the mcp-tool-secrets Kubernetes Secret the sidecar consumes. Keeping both in one module lets CI/CD pipelines render the entire pipeline atomically and guarantees the key names in SECRET_MAPPINGS stay aligned with the env_from lookups in the deployment.

Code snippetpython
1# k8s_manifests.py — MCP sidecar pod + ESO secret pipeline
2from kubernetes import client
3
4GCP_PROJECT = "genai-prod-project"
5KSA_NAME = "mcp-workload-sa"
6SECRET_NAME = "mcp-tool-secrets"
7
8def build_mcp_sidecar_deployment(
9    app_image: str,
10    mcp_image: str,
11    replicas: int = 3,
12) -> client.V1Deployment:
13    """Build a Deployment with MCP sidecar consuming ESO-managed secrets."""
14    secret_volume = client.V1Volume(
15        name="mcp-secrets",
16        secret=client.V1SecretVolumeSource(
17            secret_name=SECRET_NAME,
18            default_mode=0o400,
19        ),
20    )
21
22    mcp_sidecar = client.V1Container(
23        name="mcp-db-tool",
24        image=mcp_image,
25        command=["python", "mcp_server.py"],
26        env_from=[
27            client.V1EnvFromSource(
28                secret_ref=client.V1SecretEnvSource(name=SECRET_NAME)
29            )
30        ],
31        ports=[client.V1ContainerPort(container_port=3001)],
32        resources=client.V1ResourceRequirements(
33            requests={"cpu": "100m", "memory": "128Mi"},
34            limits={"cpu": "500m", "memory": "256Mi"},
35        ),
36        readiness_probe=client.V1Probe(
37            http_get=client.V1HTTPGetAction(path="/health", port=3001),
38            initial_delay_seconds=5,
39            period_seconds=10,
40        ),
41        volume_mounts=[
42            client.V1VolumeMount(
43                name="mcp-secrets",
44                mount_path="/etc/mcp-secrets",
45                read_only=True,
46            )
47        ],
48    )
49
50    app_container = client.V1Container(
51        name="genai-app",
52        image=app_image,
53        ports=[client.V1ContainerPort(container_port=8000)],
54        env=[
55            client.V1EnvVar(name="MCP_SERVER_URL", value="http://localhost:3001"),
56            client.V1EnvVar(name="MCP_TRANSPORT", value="sse"),
57        ],
58        resources=client.V1ResourceRequirements(
59            requests={"cpu": "500m", "memory": "512Mi"},
60            limits={"cpu": "2000m", "memory": "1Gi"},
61        ),
62        readiness_probe=client.V1Probe(
63            http_get=client.V1HTTPGetAction(path="/healthz", port=8000),
64            initial_delay_seconds=10,
65            period_seconds=15,
66        ),
67    )
68
69    return client.V1Deployment(
70        metadata=client.V1ObjectMeta(name="genai-app", labels={"app": "genai"}),
71        spec=client.V1DeploymentSpec(
72            replicas=replicas,
73            selector=client.V1LabelSelector(match_labels={"app": "genai"}),
74            template=client.V1PodTemplateSpec(
75                metadata=client.V1ObjectMeta(labels={"app": "genai"}),
76                spec=client.V1PodSpec(
77                    service_account_name=KSA_NAME,
78                    containers=[app_container, mcp_sidecar],
79                    volumes=[secret_volume],
80                ),
81            ),
82        ),
83    )
84
85SECRET_MAPPINGS = [
86    {"gcp_name": "mcp-db-host", "k8s_key": "DB_HOST"},
87    {"gcp_name": "mcp-db-password", "k8s_key": "DB_PASSWORD"},
88    {"gcp_name": "mcp-db-name", "k8s_key": "DB_NAME"},
89    {"gcp_name": "mcp-db-user", "k8s_key": "DB_USER"},
90]
91
92SECRET_STORE = {
93    "apiVersion": "external-secrets.io/v1beta1",
94    "kind": "SecretStore",
95    "metadata": {"name": "gcp-secret-store", "namespace": "genai"},
96    "spec": {
97        "provider": {
98            "gcpsm": {
99                "projectID": GCP_PROJECT,
100                "auth": {
101                    "workloadIdentity": {
102                        "clusterLocation": "us-central1",
103                        "clusterName": "genai-cluster",
104                        "serviceAccountRef": {"name": KSA_NAME},
105                    }
106                },
107            }
108        }
109    },
110}
111
112EXTERNAL_SECRET = {
113    "apiVersion": "external-secrets.io/v1beta1",
114    "kind": "ExternalSecret",
115    "metadata": {"name": "mcp-tool-external", "namespace": "genai"},
116    "spec": {
117        "refreshInterval": "1h",
118        "secretStoreRef": {"name": "gcp-secret-store", "kind": "SecretStore"},
119        "target": {
120            "name": SECRET_NAME,
121            "creationPolicy": "Owner",
122            "deletionPolicy": "Retain",
123        },
124        "data": [
125            {
126                "secretKey": m["k8s_key"],
127                "remoteRef": {"key": m["gcp_name"], "version": "latest"},
128            }
129            for m in SECRET_MAPPINGS
130        ],
131    },
132}

• Module constants (GCP_PROJECT, KSA_NAME, SECRET_NAME): single source of truth shared by the Deployment, SecretStore, and ExternalSecret. The Kubernetes ServiceAccount is bound to a GCP IAM service account via Workload Identity—ESO uses this binding to fetch secrets without JSON key files
• secret_volume: a V1Volume backed by the Kubernetes Secret that ESO populates. default_mode=0o400 sets owner-read-only file permissions, so only the container's process UID can read credentials
• mcp_sidecar: uses env_from with a V1SecretEnvSource to inject every key from mcp-tool-secrets as environment variables—the names match the os.environ lookups in mcp_server.py. Conservative resource requests (100m CPU, 128Mi memory) keep the sidecar from skewing HPA metrics. The /health probe ensures the MCP server is initialized before the pod receives traffic
• app_container: receives MCP_SERVER_URL=http://localhost:3001 (cheap because of the shared pod network namespace) and MCP_TRANSPORT="sse". Resource limits are higher because FastAPI handles request parsing, LLM orchestration, and response streaming
• V1Deployment spec: ties both containers and the secret volume to a pod template whose service_account_name is the Workload-Identity-bound KSA referenced by the SecretStore
• SECRET_MAPPINGS: declarative list mapping GCP Secret Manager names to Kubernetes Secret keys. The k8s_key values match the env-var names mcp_server.py reads, so the GCP secret names can rotate without touching the application
• SECRET_STORE: ESO's SecretStore custom resource. The gcpsm provider authenticates via workloadIdentity, eliminating JSON key files entirely—the GKE node's metadata server issues tokens
• EXTERNAL_SECRET: a one-hour refreshInterval means ESO polls GCP Secret Manager every 60 minutes for rotated values. creationPolicy: "Owner" lets ESO create the Kubernetes Secret; deletionPolicy: "Retain" keeps it alive even if someone accidentally deletes the ExternalSecret, preventing a cascading outage. The data list-comprehension generates one entry per mapping, all referencing the "latest" version (pin to specific version numbers in production for auditability)

Do's and Don'ts

Do's

1. ✓Do wire MCP sidecar credentials through external-secrets-operator with a SecretStore + ExternalSecret backed by Workload Identity, then mount the resulting mcp-tool-secrets K8s Secret as a projected volume — projected volumes deliver rotated values to the running container without a pod restart, eliminating the window where agents silently return empty results while a credential version mismatch is hunted down.
2. ✓Do enforce the ALLOWED_PREFIXES tuple ("SELECT", "WITH", "EXPLAIN") in query_database before handing any SQL to the asyncpg pool — the LLM that calls this tool can construct arbitrary strings; without the allow-list check, a tool advertised as read-only becomes an unrestricted database write path the agent can exploit inadvertently or an attacker can exploit via prompt injection.
3. ✓Do keep the MCP Server instance and its asyncpg connection pool (min_size=2, max_size=10) as the only sidecar state, and expose the tool over stdio transport via mcp.server.stdio.run_server — this keeps the sidecar horizontally scalable and limits the FastAPI container's coupling to a single localhost:3001 JSON-RPC call, so the tool server can be replaced or restarted without touching the application container.

Don'ts

1. ✗Don't hardcode database credentials or API keys in the Deployment manifest's env: block or bake them into the container image — secret rotation then requires either an image rebuild or a manifest redeploy, and the gap between the GCP Secret Manager update and the pod restart is exactly the window that triggers agent hallucinations and on-call pages described in the introduction.
2. ✗Don't expose the MCP tool server via a Kubernetes Service and route FastAPI requests across the cluster network instead of over localhost:3001 — the entire point of the sidecar pattern is that both containers share the pod's network namespace, giving you zero cross-node latency and no additional network boundary to secure; pulling them apart re-introduces the latency and forces you to manage mTLS or network policies that the localhost channel makes unnecessary.
3. ✗Don't omit the normalized.startswith(p) allow-list check or replace it with a blocklist of disallowed keywords — SQL keyword blocklists are trivially bypassed with comment injection or whitespace tricks, while the prefix-based ALLOWED_PREFIXES check on the .strip().upper() form rejects anything that isn't a read statement before it ever reaches the connection pool.