Implement tenant-aware routing with per-tenant model and guardrail config

Introduction

When you serve many customers from one AI platform, every tenant wants something different — a stricter guardrail, a private retrieval index, Opus instead of Sonnet, a tighter rate limit — and shipping the wrong policy to a regulated tenant can breach an SLA before anyone notices. A multi-tenant GenAI platform serves many customers from one deployment, but each tenant needs its own model choice, guardrails, retrieval index, eval suite, and rate limit. This lesson shows how to resolve a per-tenant config object at the request edge and thread it through one shared code path — so onboarding a new tenant is a YAML edit, not a fork — while preserving auditability via a config_version that flows into every log and trace. By the end you'll be able to model the TenantConfig schema, wire up the TenantRouter resolver, and verify config_version flows into every audit log.

Key Terminology

• TenantConfig — the frozen Pydantic object resolved per request that carries the tenant's primary model, guardrails, retrieval index, eval suite, rate limit, and config_version; every downstream component reads from this single snapshot so business logic stays tenant-agnostic.
• TenantRouter — the resolver that maps a tenant identity (signed JWT claim or X-Tenant-Id header) to a merged TenantConfig, owns the watcher that hot-reloads the config store, and refuses unknown tenants with 403 rather than silently falling back to a default.
• Default-and-override merge — field-by-field merge of _default.yaml with a tenant's YAML, so unset fields inherit fleet defaults and a fleet-wide change propagates to non-pinning tenants without per-tenant edits.
• config_version — a short SHA over the merged tenant document, attached to every log line, span, and eval run so behavior changes are traceable to the exact config that produced them.

Concepts

The single code path, many tenants principle

Every request enters the same FastAPI handler. Before the LLM call, a TenantRouter extracts the tenant identity from a signed JWT claim (or, for service-to-service calls, an X-Tenant-Id header), looks up the merged config, and attaches it to the request scope. From that point on, the model client, prompt assembler, guardrail middleware, and retrieval client all read from the same TenantConfig object. There is no tenant-specific branching in business logic — only data lookups (see Code Walkthrough).

Resolution happens once and the resulting object fans out as read-only context. Treating TenantConfig as immutable for the lifetime of a request prevents a subtle class of bugs where a config reload halfway through a streamed response swaps models mid-call.

Tiers, feature flags, and pilot rollouts

Tier is a coarse policy lever: free gets claude-haiku-default and 60 rpm, pro gets Sonnet and 600 rpm, enterprise unlocks Opus, custom retrieval indexes, and 5000 rpm. Encode tier defaults in _default.yaml variants or a tier-template loader; do not bake tier logic into the router.

For a new model rollout, add a feature flag in routing.feature_flags and let the model router consult it. Pilot tenants flip the flag in their override; everyone else continues on the previous primary. Once telemetry and eval results from pilots are clean for a release window, flip the flag default to true in _default.yaml. Tenants that opted out have already pinned false, so the fleet-wide change is safe.

Operating discipline

• Treat _default.yaml as production code. Review it in PRs, run a contract test that loads every tenant file against the new default to catch validation regressions.
• Pin the config_version of every offline eval run. Eval results without a config version cannot be compared across time.
• Reload events emit a metric and a log line; spikes mean someone is editing the ConfigMap by hand.
• A request never falls back to "default tenant" when resolution fails — it returns 403. Silent defaults hide misconfigured clients.
• The watcher debounces rapid edits (300 ms) so a multi-file kubectl apply produces one reload.

Pitfalls

• Mutating TenantConfig mid-request after a reload. Resolve once at the request edge and pass the same object down.
• Putting secrets in tenant YAML. Keep API keys in a Secret reference; the YAML carries an alias.
• Allowing tenants to override fields that are not safe to override (model names not on the approved list, raised rate limits). Validate against an allowlist in the Pydantic model.
• Forgetting that config_version must change when _default.yaml changes. The hash captures the merged document for exactly this reason.
• Rolling a new model by flipping _default.yaml directly. Always pilot through a feature flag.
• Skipping the eval suite per tier. A free-tier model that passes the enterprise eval is over-spec'd and expensive; an enterprise model that only ran the baseline eval is under-tested.

Code Walkthrough

Building on the previous section, this walkthrough demonstrates the YAML config store, the TenantConfig schema, the TenantRouter resolver, and the audit log line that stamps every LLM call with tenant_id + config_version.

Storing tenant configs and reloading them

In Kubernetes, the natural home is a ConfigMap mounted at /etc/tenants/. Each tenant gets one YAML file, and a global _default.yaml provides the baseline. A filesystem watcher (watchdog or a kube informer when the ConfigMap is huge) reloads on change without a pod restart.

Code snippetyaml
1# /etc/tenants/_default.yaml
2tier: free
3system_prompt: "You are a helpful assistant. Be concise."
4routing:
5  primary_model: claude-sonnet-default
6  fallback_models: [claude-haiku-default]
7  feature_flags:
8    enable_new_router_2026q2: false
9guardrails:
10  pii_redaction: true
11  profanity_filter: soft
12  max_output_tokens: 2048
13retrieval_index: shared-default
14eval_suite: baseline-eval-v1
15rate_limit_rpm: 60
16
17---
18# /etc/tenants/acme-health.yaml  (enterprise tenant override)
19tenant_id: acme-health
20tier: enterprise
21system_prompt: |
22  You are Acme Health's clinical-documentation assistant.
23  Never produce a diagnosis; always cite the source paragraph.
24routing:
25  primary_model: claude-opus-clinical
26  fallback_models: [claude-sonnet-default]
27guardrails:
28  blocked_topics: [recreational-drugs, self-harm-instructions]
29  profanity_filter: strict
30retrieval_index: acme-health-emr-v3
31eval_suite: clinical-doc-eval-v4
32rate_limit_rpm: 1200

The default-and-override merge is field-by-field, not document-replace. A tenant that only sets rate_limit_rpm keeps every other inherited value. This is what allows the platform team to roll a fleet-wide change to _default.yaml and have it propagate to every tenant that has not pinned the field.

Schema, resolver, and audit trail

Pydantic gives validation, defaults, and a predictable JSON-schema export for tooling; frozen=True enforces the immutability contract. The TenantRouter swaps the resolved dict atomically under an RLock so the watcher thread cannot race a request reader. Every LLM call logs tenant_id plus config_version so behavioral drift becomes a SQL query against a timestamped version rather than a forensic exercise.

Code snippetpython
1import hashlib
2import json
3from pathlib import Path
4from threading import RLock
5from typing import Literal
6
7import structlog
8import yaml
9from fastapi import HTTPException, Request
10from pydantic import BaseModel, ConfigDict, Field
11
12class GuardrailConfig(BaseModel):
13    pii_redaction: bool = True
14    blocked_topics: list[str] = Field(default_factory=list)
15    max_output_tokens: int = 2048
16    profanity_filter: Literal["off", "soft", "strict"] = "soft"
17
18class RoutingConfig(BaseModel):
19    primary_model: str = "claude-sonnet-default"
20    fallback_models: list[str] = Field(default_factory=list)
21    feature_flags: dict[str, bool] = Field(default_factory=dict)
22
23class TenantConfig(BaseModel):
24    model_config = ConfigDict(frozen=True)
25
26    tenant_id: str
27    tier: Literal["free", "pro", "enterprise"] = "free"
28    config_version: str  # SHA of the merged config; flows into traces
29    system_prompt: str
30    routing: RoutingConfig = RoutingConfig()
31    guardrails: GuardrailConfig = GuardrailConfig()
32    retrieval_index: str = "shared-default"
33    eval_suite: str = "baseline-eval-v1"
34    rate_limit_rpm: int = 60
35
36def _hash(doc: dict) -> str:
37    return hashlib.sha256(json.dumps(doc, sort_keys=True).encode()).hexdigest()[:12]
38
39class TenantRouter:
40    def __init__(self, config_dir: Path):
41        self._dir = config_dir
42        self._lock = RLock()
43        self._configs: dict[str, TenantConfig] = {}
44        self._default: dict = {}
45        self.reload()
46
47    def reload(self) -> None:
48        with self._lock:
49            self._default = yaml.safe_load((self._dir / "_default.yaml").read_text())
50            new: dict[str, TenantConfig] = {}
51            for path in self._dir.glob("*.yaml"):
52                if path.name.startswith("_"):
53                    continue
54                raw = yaml.safe_load(path.read_text())
55                merged = _deep_merge(self._default, raw)
56                merged["config_version"] = _hash(merged)
57                new[merged["tenant_id"]] = TenantConfig(**merged)
58            self._configs = new
59
60    def resolve(self, request: Request) -> TenantConfig:
61        tenant_id = _extract_tenant_id(request)  # JWT claim or X-Tenant-Id header
62        with self._lock:
63            cfg = self._configs.get(tenant_id)
64        if cfg is None:
65            raise HTTPException(status_code=403, detail="unknown tenant")
66        return cfg
67
68log = structlog.get_logger()
69
70async def llm_call(request: Request, body: dict, router: TenantRouter):
71    cfg = router.resolve(request)
72    response = await gateway.invoke(
73        model=cfg.routing.primary_model,
74        system=cfg.system_prompt,
75        messages=body["messages"],
76        guardrails=cfg.guardrails,
77    )
78    log.info(
79        "llm_call",
80        tenant_id=cfg.tenant_id,
81        config_version=cfg.config_version,
82        model=cfg.routing.primary_model,
83        eval_suite=cfg.eval_suite,
84        tier=cfg.tier,
85    )
86    return response

_deep_merge walks both documents and prefers the override at every leaf — implement it explicitly rather than relying on dict.update, which clobbers nested keys. You'll know it works when a tenant-scoped request shows the merged primary_model in the structured log alongside a config_version that changes the moment you edit and reload the ConfigMap.

Do's and Don'ts

Do's

1. ✓Do implement _deep_merge explicitly — walking both documents and preferring the override at every leaf — this is what allows a tenant YAML that sets only rate_limit_rpm to automatically inherit fleet-wide changes to system_prompt, guardrails, or routing.fallback_models from _default.yaml; without it, a field the tenant never touched can silently disappear.
2. ✓Do compute config_version as hashlib.sha256(json.dumps(merged, sort_keys=True)) on the fully merged dict after _deep_merge, not on the raw tenant file — the hash must capture the effective policy in force, so a _default.yaml change that propagates to a tenant who hasn't pinned the field registers as a new config_version in every structlog line and OpenTelemetry span, making behavioral drift a SQL query rather than a forensic exercise.
3. ✓Do declare TenantConfig with ConfigDict(frozen=True) and hold the RLock only for the atomic self._configs = new swap inside reload() — the frozen object guarantees no downstream component (prompt builder, guardrails, retriever) can mutate the config after resolve() hands it off, and the narrow lock window lets a watchdog filesystem reload proceed without blocking concurrent request readers.

Don'ts

1. ✗Don't use dict.update as the default-override merge strategy — it performs a shallow merge, so a tenant YAML that only overrides routing.primary_model silently clobbers the entire routing sub-document, wiping fallback_models and feature_flags inherited from _default.yaml with no validation error from Pydantic.
2. ✗Don't compute config_version before _deep_merge runs on the raw tenant override YAML — a pre-merge hash stays constant when _default.yaml changes, so tenants that inherit the updated default appear unchanged in traces; blocked_topics guardrails or eval_suite changes propagated through the default become invisible in the audit log until a tenant explicitly edits their own file.
3. ✗Don't silently return a fallback config when resolve() gets an unknown tenant_id — raise HTTPException(status_code=403, detail="unknown tenant") instead; routing an unrecognized client through _default.yaml policy risks applying free-tier rate_limit_rpm: 60, skipping enterprise blocked_topics guardrails like recreational-drugs or self-harm-instructions, and borrowing shared-default retrieval instead of the tenant's private EMR index.