Build multi-server MCP clients

Introduction

When an agent needs to query a filesystem server, a database server, and an external API server simultaneously, connecting to each one ad hoc produces tangled setup code, colliding tool names, and no clear mechanism for routing a call to the right server. Without a structured approach, each new server multiplies the complexity. In this lesson you will build a MultiServerManager that registers MCP servers declaratively, opens all connections concurrently with asyncio.gather, namespaces tools by server ID to eliminate name collisions, and routes tool calls through a central index — so your agent can treat dozens of servers as a single unified tool surface.

Key Terminology

• ServerRegistration — A dataclass that holds the declarative configuration for one MCP server (server_id, config, optional tags) before any network connection is made; it is the pure-data input written by register_server into self.servers.
• ManagedServer — A dataclass that pairs a ServerRegistration with live connection state (connection, session) and the list of tools discovered from that server after connect_all has run.
• MultiServerManager — The orchestrating dataclass that owns the registry of ServerRegistration entries, the dictionary of ManagedServer instances, and the _tool_index that routes any tool call to the correct server in O(1).
• Tool namespacing — The practice of prefixing each discovered tool name with its originating server_id (e.g., "filesystem.read_file") so that identically-named tools from different servers never collide on the manager's shared tool surface.
• _tool_index — A dictionary inside MultiServerManager that maps every namespaced tool name to the server_id that owns it, enabling constant-time routing of agent tool calls without scanning the full set of managed servers.
• connect_all — The async method that fires every registered server's connection attempt concurrently via asyncio.gather, collapsing total startup time from the sum of all handshake durations to the duration of the single slowest server.

Concepts

Separating Configuration from Connection

A persistent source of complexity in multi-server clients is conflating two distinct concerns: what servers exist and when to open connections to them. The MultiServerManager resolves this with a deliberate two-phase model. register_server is a synchronous, side-effect-free call — it writes a ServerRegistration into self.servers and returns immediately without touching any network. Only when connect_all is explicitly invoked does the manager enter the effectful IO phase.

This separation matters beyond style. It lets the application accumulate a complete server roster through configuration parsing or dependency injection before any connection is attempted. It also makes the IO phase a batchable unit: all connection work happens in one place, under one coroutine, where it can be optimized as a group rather than scattered across the code wherever a server happens to be registered.

Concurrent Connection with asyncio.gather

Connecting to N MCP servers serially means total startup latency is the sum of all N handshake durations. For a 10-server configuration where each handshake takes 100–200 ms, that is 1–2 seconds of dead time before the agent can route a single call.

connect_all eliminates this by passing a generator of connect_one coroutines directly to asyncio.gather. All N handshakes run concurrently; the wait collapses to the duration of the single slowest server regardless of how many others are registered. After connect_all returns, the assertion len(manager._managed) == len(manager.servers) provides a tight health check — matching counts confirm every registered server produced a live ManagedServer entry before any tool call is routed (see Code Walkthrough).

Tool Namespacing and O(1) Routing

Without namespacing, two servers that both expose a tool named search or read_file force the manager to either silently drop one or resolve conflicts at call time — both outcomes are fragile. Tool namespacing eliminates the collision entirely: each tool is stored under the key "{server_id}.{tool_name}", making every name globally unique across the unified surface.

The _tool_index dictionary is the architectural payoff: any tool call arriving from the agent is dispatched in O(1) by a single dictionary lookup rather than iterating over _managed. The agent sees one flat namespace of tool names and needs no knowledge of which underlying server owns each one — the manager's routing layer is fully transparent.

Code Walkthrough

Now that you've seen Separating Configuration from Connection, Concurrent Connection with asyncio.gather, and Tool Namespacing and O(1) Routing, this walkthrough turns them into working code.

The implementation rests on three dataclasses that separate concerns cleanly. ServerRegistration holds the declarative configuration for a server before any connection is made. ManagedServer pairs that registration with the live connection state and the list of tools discovered from that server. MultiServerManager owns dictionaries for both and a _tool_index that maps each namespaced tool name to its originating server ID, enabling O(1) routing at call time.

Code snippetpython
1import asyncio
2from dataclasses import dataclass, field
3from typing import Dict, List, Optional
4from mcp import ClientSession
5
6@dataclass
7class ServerRegistration:
8    server_id: str          # unique key used for namespacing
9    config: MCPClientConfig
10    tags: List[str] = field(default_factory=list)
11
12@dataclass
13class ManagedServer:
14    registration: ServerRegistration
15    connection: Optional[MCPConnection] = None
16    session: Optional[ClientSession] = None
17    tools: List[ToolDefinition] = field(default_factory=list)
18
19@dataclass
20class MultiServerManager:
21    servers: Dict[str, ServerRegistration] = field(default_factory=dict)
22    _managed: Dict[str, ManagedServer] = field(default_factory=dict)
23    _tool_index: Dict[str, str] = field(default_factory=dict)  # tool_name → server_id

With the data model in place, register_server writes a ServerRegistration entry into self.servers without opening any connection. connect_all then fires every connection attempt in parallel using asyncio.gather, reducing startup time from an O(N) serial chain of handshakes to the duration of the single slowest server response.

Code snippetpython
1    def register_server(
2        self,
3        server_id: str,
4        config: MCPClientConfig,
5        tags: Optional[List[str]] = None,
6    ) -> None:
7        self.servers[server_id] = ServerRegistration(
8            server_id=server_id,
9            config=config,
10            tags=tags or [],
11        )
12
13    async def connect_all(self) -> Dict[str, ClientSession]:
14        sessions: Dict[str, ClientSession] = {}
15
16        async def connect_one(server_id: str, reg: ServerRegistration) -> None:
17            connection = MCPConnection(reg.config)
18            managed = ManagedServer(registration=reg, connection=connection)
19            self._managed[server_id] = managed
20            sessions[server_id] = managed.session
21
22        await asyncio.gather(
23            *(connect_one(sid, reg) for sid, reg in self.servers.items())
24        )
25        return sessions

After calling connect_all(), verify the manager is healthy by asserting len(manager._managed) == len(manager.servers) — matching counts confirm that every registered server successfully established a managed connection before your agent begins routing tool calls.

Do's and Don'ts

Having walked through building multi-server clients above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do call connect_all() with asyncio.gather — firing every MCP handshake concurrently collapses what would otherwise be an O(N) serial chain down to the latency of the single slowest server; when managing a filesystem server, a database server, and an API server at once, that difference grows with each additional registration.
2. ✓Do namespace every tool name with its server_id in _tool_index — two servers that both expose a tool named query or read_file produce silent collisions without the prefix; the index maps each namespaced name to the originating server_id for O(1) routing at call time, which is the only mechanism the manager has for dispatching the right call to the right server.
3. ✓Do assert len(manager._managed) == len(manager.servers) immediately after connect_all() — matching counts confirm that every ServerRegistration produced a live ManagedServer entry before your agent begins routing tool calls; a mismatch means at least one server silently failed to connect and any tool call routed to it will misfire.

Don'ts

1. ✗Don't open connections inside register_server — register_server is intentionally a declarative step that writes a ServerRegistration into self.servers without touching the network; embedding connection logic there collapses the clean boundary between configuration state (ServerRegistration) and live runtime state (ManagedServer), making it impossible to batch connections under asyncio.gather.
2. ✗Don't route tool calls by iterating through _managed at dispatch time — the _tool_index dict exists precisely to replace that O(N) scan with O(1) lookup from a namespaced tool name to its originating server_id; bypassing the index means every tool dispatch slows linearly as new servers are registered.
3. ✗Don't reuse the same server_id across register_server calls — self.servers is a dict keyed by server_id, so a duplicate silently overwrites the prior ServerRegistration and any tools already indexed under that namespace in _tool_index will route to whichever server registered last rather than the intended one.