LiteLLM Gateway

You will deploy LiteLLM as a production AI gateway in K8s using the official Helm chart. Configure it to route to OpenAI Responses API, Anthropic Messages API, and Gemini API using `model_list` configuration. Set up virtual keys per department via `/key/generate` endpoint, RBAC roles with model-level access controls, and rate limits per key to enforce per-department usage quotas.

You will configure LiteLLM failover chains: if OpenAI returns errors, route to Anthropic; if Anthropic is down, route to Gemini, using `fallbacks` configuration. Implement circuit breakers via `allowed_fails` and `cooldown_time` that automatically disable a provider after N consecutive failures and re-enable after a cooldown period. Test by simulating provider outages and verifying automatic failover with measured recovery times.

You will build `GatewayLoadTester` that stress-tests your LiteLLM deployment under increasing concurrent request loads. Measure throughput, latency percentiles (P50, P95, P99), error rates, and failover trigger points at 10, 50, 100, and 500 concurrent requests. Implement `plan_capacity()` that, given projected request volumes, recommends LiteLLM replica count, rate limit settings, and provider quota allocations. Output a capacity plan document.

You will build a comprehensive testing framework for litellm gateway. Implement unit tests for each component, integration tests that verify end-to-end behavior, and regression tests using golden test sets. Build a test runner that executes all test types, reports results with pass/fail per test case, and tracks test coverage. Implement chaos testing scenarios that verify system resilience under failure conditions.

You will optimize litellm gateway for production scale. Profile the system under realistic load to identify bottlenecks. Implement caching, connection pooling, and async processing where applicable. Build autoscaling configuration based on traffic patterns. Measure and reduce resource consumption while maintaining SLA targets. Target handling 10x current load with < 20% latency increase.

You will build operational documentation and runbooks for litellm gateway. Document the architecture, configuration reference, and troubleshooting guide. Build executable runbook steps for the top 5 failure scenarios specific to this system. Implement health check and readiness endpoints. Create a monitoring dashboard showing key operational metrics with alerting thresholds.