Deploy multimodal defense on GKE with resource limits

Introduction

When you deploy multimodal LLM applications to Kubernetes, you need a defense layer that sits close to the application without complicating your main service. The sidecar pattern solves this: a FastAPI process runs in the same pod, intercepts multimodal inputs over localhost, and applies hosted-API-backed scanning before any content reaches the LLM backend. By the end of this lesson, you'll be able to write the FastAPI entry point for a multimodal defense sidecar and understand how it routes scan requests to hosted vision and audio APIs through LiteLLM.

Key Terminology

• Sidecar Pattern — an architectural pattern where a secondary process (here, a FastAPI service) runs in the same Kubernetes pod as the main application, communicating over localhost to intercept and validate inputs before they reach the LLM backend.
• Multimodal Scan Request — the structured payload accepted by the /scan endpoint, represented by MultimodalScanRequest, which carries any combination of text, image_b64, and audio_b64 fields for cross-modal validation.
• Aggregate Risk Score — a numeric value returned by ScanResult that quantifies the combined threat level across all submitted modalities; a decision of "block" causes the sidecar to raise HTTP 422.
• Cross-Modal Conflict — a discrepancy detected between modalities (e.g., image text contradicting the accompanying caption) surfaced as entries in ScanResult.cross_modal_conflicts and reported in the 422 error detail.
• Vision Routing Config — a per-scan-type mapping (VISION_ROUTING_CONFIG) that assigns a primary hosted model, a fallback model, and a latency budget; LiteLLM uses this to route between providers and activate the fallback automatically on timeout or outage.
• Hosted Vision and Audio APIs — external provider endpoints (GPT-4o, GPT-4o-mini, Gemini 1.5 Pro/Flash, Whisper) called through LiteLLM instead of local GPU inference, eliminating the need for GPU node pools in the Kubernetes cluster.

Concepts

Why a Sidecar Instead of an Inline Filter

The central architectural choice in this lesson is placing defense logic in a separate process that shares the pod's network namespace rather than embedding it directly in the application service. The sidecar runs its own FastAPI process and listens on a localhost port; the main application sends every multimodal payload there before forwarding to the LLM backend. This separation of concerns means the main service never needs to know which scanning models are in use, and the sidecar can be updated, restarted, or replaced independently — a critical property when hosted API providers change their model versions or when you need to tighten scanning rules without redeploying the entire application.

Kubernetes amplifies this benefit: the /health endpoint the sidecar exposes lets liveness and readiness probes monitor the scanner independently of the application container. If the sidecar becomes unhealthy, Kubernetes restarts that container only — not the full pod. The tradeoff is that every request incurs a localhost round-trip and a hosted-API call, so the routing configuration's latency budgets (max_latency_ms) are a real operational constraint, not optional metadata (see Code Walkthrough).

Hosted APIs as a Substitute for GPU Inference

Running vision or audio models locally would require GPU node pools — expensive, hard to autoscale, and operationally complex. By routing all multimodal scanning through LiteLLM to hosted endpoints (GPT-4o, Gemini 1.5 Pro, Whisper), the sidecar offloads GPU compute entirely. The application cluster needs only CPU nodes, and capacity scales with API quota rather than cluster size.

LiteLLM's provider-agnostic interface is what makes this practical: VISION_ROUTING_CONFIG names models from different providers in the same dictionary, and the fallback entry activates transparently when the primary provider exceeds its latency threshold or returns an error. This means a single-provider outage doesn't block the entire defense layer — the sidecar keeps scanning using the fallback model.

Scan Tiers and Latency Budgets

Not every scan operation warrants the same model or the same wait time. OCR extraction — reading text embedded in an image — is cheaper and faster with a smaller model (gpt-4o-mini, 5-second budget). Adversarial perturbation detection — identifying pixel-level manipulations designed to mislead the LLM — demands a more capable model (gpt-4o, 10-second budget). The VISION_ROUTING_CONFIG dictionary makes this tiering explicit and configurable without changing application code.

When all modalities are present in a single request, the sidecar must aggregate risk scores across text, image, and audio channels. Cross-modal conflicts — cases where one modality appears to contradict or subvert another — are surfaced explicitly in ScanResult.cross_modal_conflicts so downstream logging and alerting can act on them, not just on the binary allow/block decision (see Code Walkthrough).

Code Walkthrough

Now that you understand the sidecar architecture and how hosted vision and audio APIs replace local GPU inference, you can see exactly how those ideas materialize in code.

FastAPI Sidecar Entry Point

The sidecar exposes two endpoints: /scan for multimodal input validation and /health for Kubernetes liveness and readiness probes. The example below is self-contained — MultimodalScanRequest and its fields are defined inline so you can run it directly.

Code snippetpython
1from fastapi import FastAPI, HTTPException
2from pydantic import BaseModel
3from typing import Optional
4
5app = FastAPI(title="Multimodal Defense Sidecar")
6
7class MultimodalScanRequest(BaseModel):
8    text: Optional[str] = None
9    image_b64: Optional[str] = None
10    audio_b64: Optional[str] = None
11
12class ScanResult(BaseModel):
13    decision: str          # "allow" or "block"
14    aggregate_risk_score: float
15    cross_modal_conflicts: list[str]
16
17async def run_scan(request: MultimodalScanRequest) -> ScanResult:
18    # In production this calls LiteLLM-backed vision/audio/text scanners.
19    # Here we return a safe default so the endpoint contract is testable.
20    return ScanResult(decision="allow", aggregate_risk_score=0.0, cross_modal_conflicts=[])
21
22@app.post("/scan", response_model=ScanResult)
23async def scan_multimodal_input(request: MultimodalScanRequest):
24    result = await run_scan(request)
25    if result.decision == "block":
26        raise HTTPException(
27            status_code=422,
28            detail={
29                "error": "multimodal_injection_detected",
30                "risk_score": result.aggregate_risk_score,
31                "conflicts": result.cross_modal_conflicts,
32            },
33        )
34    return result
35
36@app.get("/health")
37async def health_check():
38    return {"status": "healthy"}

Hosted Vision API Routing Configuration

The sidecar selects which hosted vision model to call based on scan type. Cheaper, faster models handle routine OCR extraction; the most capable models are reserved for adversarial perturbation detection. LiteLLM routes between providers and automatically falls back when a provider exceeds the latency threshold.

Code snippetpython
1VISION_ROUTING_CONFIG = {
2    "ocr_extraction": {
3        "primary": "gpt-4o-mini",
4        "fallback": "gemini/gemini-1.5-flash",
5        "max_latency_ms": 5000,
6    },
7    "perturbation_detection": {
8        "primary": "gpt-4o",
9        "fallback": "gemini/gemini-1.5-pro",
10        "max_latency_ms": 10000,
11    },
12}

ocr_extraction uses GPT-4o-mini as the primary model (lower cost, sufficient for text extraction) with Gemini 1.5 Flash as the fallback and a 5-second budget. perturbation_detection escalates to GPT-4o with a 10-second budget, matching the higher compute demand of adversarial image analysis. LiteLLM's provider-agnostic interface means the sidecar keeps working even during a single-provider outage — the fallback entry activates transparently.

Confirm that the /scan endpoint returns HTTP 200 for clean inputs and HTTP 422 with "error": "multimodal_injection_detected" in the response body when decision is "block". You'll know it works when Kubernetes liveness probes on /health return {"status": "healthy"} and your application pod logs show the sidecar intercepting requests on localhost before they reach the LLM backend.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do expose both /scan and /health endpoints in the FastAPI sidecar — Kubernetes liveness and readiness probes target /health; omitting it causes the pod to restart-loop even when the sidecar is correctly intercepting requests on localhost, taking your defense layer down with it.
2. ✓Do tier VISION_ROUTING_CONFIG by scan type — assign gpt-4o-mini (with gemini/gemini-1.5-flash as fallback) to ocr_extraction and escalate to gpt-4o (with gemini/gemini-1.5-pro) for perturbation_detection; matching model capability to threat complexity keeps latency within the 5-second and 10-second budgets respectively without overspending compute on routine text extraction.
3. ✓Do raise HTTP 422 with the full structured detail body ("error": "multimodal_injection_detected", "risk_score", "conflicts") when decision == "block" — the main service disambiguates a detected injection from an unexpected sidecar crash using that specific status code and error key; returning a bare 500 or 400 breaks the interception contract silently.

Don'ts

1. ✗Don't omit the fallback entry in VISION_ROUTING_CONFIG — without a fallback model (e.g., dropping "fallback": "gemini/gemini-1.5-flash" from ocr_extraction), a single-provider outage disables the entire scan path and every multimodal input either hard-errors or passes unscanned; LiteLLM's automatic failover only activates when the fallback key is present.
2. ✗Don't mark text, image_b64, or audio_b64 required in MultimodalScanRequest — the Optional typing is load-bearing; real-world traffic sends text-only, image-only, or audio-only payloads, and any required field causes Pydantic to reject those requests before run_scan is ever called, silently disabling the sidecar for partial-modality inputs.
3. ✗Don't route the sidecar's /scan endpoint through the Kubernetes ingress — the sidecar is designed to accept calls only over localhost within the pod; exposing it externally creates an unauthenticated surface where attackers can probe your injection detector with arbitrary payloads without touching the main application at all.