Build MCP tool routing with load balancing and failover

Introduction

When you deploy dozens of MCP servers and a single tool invocation has five viable replicas, every call becomes a routing decision — and getting it wrong means cascading timeouts, hot-spotted servers, and agent retries that burn context tokens on errors instead of useful work. By the end of this lesson you'll be able to architect a tool router that pre-filters candidates on capability, authorization, resource class, and region locality, scores the survivors by active load and P95 latency, and fails over cleanly to the next-best replica when one trips its circuit breaker.

Key Terminology

• Tool router: the infrastructure layer that selects which MCP server replica handles a given tool invocation, applying pre-filtering, scoring, and failover across candidates.
• Pre-filter: the first routing stage that eliminates replicas that cannot or should not serve a request — based on capability match, authorization policy, and circuit-breaker health — before any scoring runs.
• Circuit breaker: a per-replica state machine (closed / open / half-open) that takes a failing server out of rotation after a consecutive-failure threshold and probes it after a cooldown before restoring traffic.
• P95 latency: the 95th-percentile response time over a rolling window, used as a load-balancing signal so that consistently slow replicas are scored down even when their connection counts are low.
• Failover: the retry loop that, on invocation failure, excludes the failed replica, re-runs the pre-filter and scoring pipeline, and dispatches to the next-best candidate.

Concepts

Intelligent pre-filtering strategies

A minimal pre-filter checks capability and circuit-breaker state (the concrete implementation appears in the Code Walkthrough), but production deployments need richer filtering. Three additional pre-filtering dimensions prove essential in governed MCP meshes:

• Authorization-aware filtering: Before scoring, the router queries the tool authorization policy engine to verify the requesting agent's roles permit access to the tool on the candidate server. This prevents routing to a server in a restricted security zone when the agent lacks the required clearance. The check is inexpensive — it reads from cached policy decisions — but it eliminates entire classes of authorization failures that would otherwise manifest as cryptic permission-denied errors at the MCP protocol layer.

• Resource-class filtering: Not all replicas are equal. A GPU-equipped server running an embedding tool can handle semantic_search calls that a CPU-only replica cannot. The router should maintain a resource_tags set on each replica and match it against tool-level resource requirements from the tool registry. A tool registered with requires: {"gpu"} would automatically exclude replicas whose resource_tags lack that entry.

• Locality-aware filtering: In multi-region deployments, routing a tool call from a US-East agent to an EU-West server adds 100+ milliseconds of latency. Pre-filtering by region affinity — preferring same-region replicas and only falling back to cross-region when local capacity is exhausted — keeps latency predictable without sacrificing availability.

These filters compose cleanly because each one independently narrows the candidate list. The order matters for performance: apply the cheapest filter first (capability set membership is an O(1) set lookup) and the most expensive last (authorization policy queries may involve cache misses).

Load balancing strategies in practice

The _compute_score method implements a least-connections-with-latency-penalty strategy, but operators need the flexibility to switch strategies based on workload characteristics:

• Weighted round-robin works best when tool execution times are uniform. Each replica receives requests proportional to its weight, cycling in a deterministic order. This strategy excels for lightweight, stateless tools like format_code or validate_json where per-request latency variance is negligible.

• Least connections (the default above) adapts dynamically to heterogeneous execution times. A server processing a long-running full_project_analysis tool naturally accumulates fewer new requests because its connection count stays elevated. This self-balancing property makes it the safest default for mixed-workload meshes.

• Latency-optimized routing tracks each replica's recent response times and aggressively favors the fastest backend. This strategy is ideal when replicas have varying hardware or when network conditions differ across zones. However, it can cause thundering herd problems: if one replica reports low latency during a quiet period, all agents simultaneously route to it, creating a spike. The latency penalty in _compute_score mitigates this by blending latency with connection count.

Circuit breaker tuning and observability

The circuit breaker's failure_threshold and cooldown duration are the two most impactful tuning knobs. Setting the threshold too low (e.g., 1) causes a single transient error to take a server offline for the full cooldown period, reducing mesh capacity unnecessarily. Setting it too high (e.g., 20) means the router continues sending requests to a genuinely broken server, wasting agent time and burning through retry budgets. A threshold of 5 consecutive failures with a 30-second cooldown represents a balanced starting point for most deployments.

Observability is non-negotiable. Every routing decision should emit structured telemetry:

• Routing events: tool name, selected server ID, candidate count, computed score, attempt number
• Failover events: tool name, failed server ID, error class, remaining candidates
• Circuit transitions: server ID, old state, new state, consecutive failure count

These events feed directly into the ecosystem governance dashboards covered in another goal, enabling operators to identify consistently failing servers, detect load imbalances, and validate that routing strategies match actual traffic patterns. The tool routing layer is where abstract governance policies become concrete, measurable operational behaviors — every routing decision is a data point that either confirms or challenges the mesh's architectural assumptions.

Code Walkthrough

Building on the pre-filtering strategies above, the router needs a concrete data model and a routing method that ties capability, authorization, resource class, and load signals into one pipeline. We start with a ServerReplica dataclass that carries everything the router scores on — the capability set, a resource_tags set for resource-class matching, a region for locality, a rolling latency window for P95, and a circuit-breaker timestamp for health gating.

Code snippetpython
1import time
2from dataclasses import dataclass, field
3
4@dataclass
5class ServerReplica:
6    server_id: str
7    capabilities: set[str] = field(default_factory=set)
8    resource_tags: set[str] = field(default_factory=set)
9    region: str = "us-east"
10    active_connections: int = 0
11    latency_window: list[float] = field(default_factory=list)
12    circuit_open_until: float = 0.0
13
14    @property
15    def is_healthy(self) -> bool:
16        return time.time() >= self.circuit_open_until
17
18    @property
19    def p95_latency(self) -> float:
20        if not self.latency_window:
21            return 0.0
22        ranked = sorted(self.latency_window)
23        idx = min(int(len(ranked) * 0.95), len(ranked) - 1)
24        return ranked[idx]

The pre-filter applies all four dimensions in turn — capability match, circuit-breaker health, the authorization check against cached policy, and the resource-class subset test — then prefers same-region replicas while falling back to any candidate. Scoring combines least-connections with a P95 latency penalty so consistently slow replicas are dispatched to less often. The route loop retries down the sorted pool, tripping the breaker on failure so the next iteration fails over.

Code snippetpython
1def pre_filter(replicas, tool, requires, agent_roles, allowed, region):
2    candidates = []
3    for r in replicas:
4        if tool not in r.capabilities or not r.is_healthy:
5            continue
6        if not allowed(agent_roles, tool, r.server_id):  # authorization-aware
7            continue
8        if not requires <= r.resource_tags:              # resource-class
9            continue
10        candidates.append(r)
11    local = [r for r in candidates if r.region == region]
12    return local or candidates                            # locality-aware
13
14def score(r):
15    return r.active_connections + r.p95_latency * 0.5
16
17def route(replicas, tool, requires, agent_roles, allowed, region, invoke):
18    pool = pre_filter(replicas, tool, requires, agent_roles, allowed, region)
19    for replica in sorted(pool, key=score):
20        replica.active_connections += 1
21        start = time.time()
22        try:
23            return invoke(replica, tool)
24        except Exception:
25            replica.circuit_open_until = time.time() + 30.0  # trip + failover
26        finally:
27            replica.latency_window.append(time.time() - start)
28            replica.active_connections -= 1
29    raise RuntimeError(f"no healthy replica for {tool}")

Each stage stays swappable — drop in a cost-optimized score without touching the filter. Verify by routing a call whose required resource tag only one replica carries, confirming the router selects that replica, then forcing it to raise and checking the next call fails over to a different candidate.

Do's and Don'ts

Do's

1. ✓Do enforce all four pre_filter gates — capability membership, is_healthy circuit-breaker check, authorization, and requires <= r.resource_tags subset test — before any replica enters the scoring step — a replica that passes only three of the four can still appear at the top of the sorted pool, causing the router to dispatch to an unauthorized, resource-mismatched, or circuit-open server before it ever tries a healthy one.
2. ✓Do score replicas with both active_connections and p95_latency * 0.5 together rather than either signal alone — least-connections alone promotes replicas that are idle because they are chronically slow, burning context tokens on high-latency responses; adding the P95 penalty demotes consistently slow servers without excluding them when all remaining candidates are under load.
3. ✓Do keep pre_filter and score as independently swappable functions rather than inlining the scoring logic inside the filter loop — the walkthrough's two-stage design lets you substitute a cost-optimized or region-weighted score without touching the capability, authorization, or resource-class gate logic, and lets you unit-test each stage against a fixed replica pool in isolation.

Don'ts

1. ✗Don't place replica.circuit_open_until = time.time() + 30.0 in the finally block instead of the except block — moving the breaker-trip to finally trips it on every call, including successful ones, so every replica cycles into a 30-second cooldown window after its first successful dispatch and the router exhausts the pool even when all replicas are healthy.
2. ✗Don't apply the locality preference after scoring — computing score(r) across the full candidate set before running local = [r for r in candidates if r.region == region] lets a cross-region replica with a momentarily low connection count outscore a same-region replica and win the dispatch, adding unnecessary cross-region latency that then feeds back into latency_window and degrades future P95 scores for the entire pool.
3. ✗Don't use an unbounded latency_window list with append and no eviction — as shown, every call appends one float and the list grows unbounded for the server's lifetime; on a long-lived replica the p95_latency property's sorted() call begins to reflect historical latency from hours ago rather than current performance, causing the score to misrank replicas and mute the router's ability to detect a degraded server in real time.