Build embedding pipelines with LiteLLM gateway routing

Introduction

When you hard-code a single embedding provider into your ingestion code, every business decision — swapping to a cheaper model, failing over during a vendor outage, A/B testing a new release — turns into a code change, a redeploy, and a regression test. Teams that wire OpenAI directly into pipelines routinely stay stuck paying premium per-token rates because the cost of migrating exceeds the savings. By the end of this lesson you will be able to build an embedding pipeline that routes calls through a LiteLLM gateway, swaps providers via configuration, picks the cheapest viable model per request, and fails over automatically when a provider degrades.

Key Terminology

• LiteLLM gateway — a translation layer that exposes one OpenAI-shaped API and dispatches each call to whichever supported provider is named in the model id; the unit you wrap your pipeline around so provider choice becomes runtime configuration.
• Provider-prefixed model id — strings like openai/text-embedding-3-large or cohere/embed-english-v4.0 that LiteLLM uses to pick the backend; the only thing that has to change to switch vendors.
• Cost-based routing — selecting a model per request by pricing the input tokens against LiteLLM's model_cost registry; keeps bulk embedding workloads on the cheapest viable backend without code changes.
• Failover chain — an ordered list of model ids the gateway iterates through on exception; turns transient provider faults into a retried call rather than a halted pipeline.
• Batch embedding — submitting a list of texts in one aembedding call instead of per-item; the single biggest throughput lever for ingestion pipelines.

Concepts

Gateway Architecture

The gateway sits between your application and embedding providers, translating a single API call into provider-specific requests. The key design decision is whether to run LiteLLM as a sidecar, a shared service, or a standalone proxy. For embedding pipelines processing millions of chunks, a shared service deployment with connection pooling provides the best balance of resource efficiency and fault isolation. A single EmbeddingGateway class wraps LiteLLM's aembedding call behind a clean async interface and accepts a provider-prefixed model identifier, so callers swap providers by changing a configuration string rather than modifying application code (see Code Walkthrough).

Cost-Based Routing

Cost-based routing directs embedding requests to the cheapest available provider that meets quality requirements. The router queries LiteLLM's model_cost registry for real-time pricing, multiplies the per-token price by the input's token count, and filters out models whose projected cost exceeds a configurable max_cost_per_request threshold. This ensures bulk workloads automatically flow to the cheapest viable provider while quality-sensitive requests can override routing with an explicit model parameter (see Code Walkthrough).

Provider Failover

Provider failover ensures pipeline reliability when a primary provider experiences outages, rate limiting, or degraded performance. The failover chain iterates through a prioritized list of models inside a try/except loop; when the current model raises any exception (timeout, 429, 5xx), the gateway logs the failure and immediately retries with the next model. A max_retries parameter controls how many times each individual model is attempted before moving on. Pairing failover with cost routing yields a gateway that is both cheap on the happy path and resilient on the unhappy one.

Batch Embedding

Production pipelines process documents in batches, not one at a time. Submitting a list of texts in a single aembedding call maximizes throughput while respecting provider rate limits and returning consistent results. A batch_size around 100 typically balances per-call overhead (too small) against timeout risk (too large) for most embedding providers, and per-batch cost is amortized evenly across items for downstream tracking.

Code Walkthrough

The snippet below combines the four concepts above into a single gateway class: select_model implements cost-based routing, embed wraps the routed call in a failover chain, and embed_batch chunks lists for high-throughput ingestion.

Code snippetpython
1import asyncio
2import logging
3from dataclasses import dataclass
4from typing import Optional
5
6import litellm
7import tiktoken
8
9logger = logging.getLogger(__name__)
10
11@dataclass
12class EmbeddingResult:
13    embedding: list[float]
14    model: str
15    provider: str
16    token_count: int
17    cost_usd: float
18
19class EmbeddingGateway:
20    def __init__(
21        self,
22        models: list[str],
23        max_cost_per_request: float = 0.01,
24        max_retries: int = 2,
25        timeout: int = 30,
26        batch_size: int = 100,
27    ):
28        self.models = models
29        self.max_cost = max_cost_per_request
30        self.max_retries = max_retries
31        self.timeout = timeout
32        self.batch_size = batch_size
33        self.encoder = tiktoken.get_encoding("cl100k_base")
34
35    def select_model(self, text: str) -> str:
36        token_count = len(self.encoder.encode(text))
37        priced: list[tuple[str, float]] = []
38        for model in self.models:
39            per_token = litellm.model_cost.get(model, {}).get(
40                "input_cost_per_token", 0
41            )
42            estimated = per_token * token_count
43            if estimated <= self.max_cost:
44                priced.append((model, estimated))
45        if not priced:
46            return self.models[0]
47        priced.sort(key=lambda item: item[1])
48        return priced[0][0]
49
50    async def embed(
51        self, text: str, model: Optional[str] = None
52    ) -> EmbeddingResult:
53        primary = model or self.select_model(text)
54        chain = [primary] + [m for m in self.models if m != primary]
55        last_error: Optional[Exception] = None
56        for candidate in chain:
57            for attempt in range(self.max_retries):
58                try:
59                    response = await litellm.aembedding(
60                        model=candidate,
61                        input=[text],
62                        timeout=self.timeout,
63                    )
64                    cost = litellm.completion_cost(
65                        completion_response=response
66                    )
67                    return EmbeddingResult(
68                        embedding=response.data[0]["embedding"],
69                        model=response.model,
70                        provider=candidate.split("/")[0],
71                        token_count=response.usage.total_tokens,
72                        cost_usd=cost,
73                    )
74                except Exception as exc:
75                    last_error = exc
76                    logger.warning(
77                        "Model %s attempt %d failed: %s",
78                        candidate,
79                        attempt + 1,
80                        exc,
81                    )
82                    await asyncio.sleep(0.5 * (attempt + 1))
83        raise RuntimeError(f"All models failed. Last error: {last_error}")
84
85    async def embed_batch(self, texts: list[str]) -> list[EmbeddingResult]:
86        results: list[EmbeddingResult] = []
87        for i in range(0, len(texts), self.batch_size):
88            batch = texts[i : i + self.batch_size]
89            model = self.select_model(batch[0])
90            response = await litellm.aembedding(
91                model=model, input=batch, timeout=self.timeout
92            )
93            cost = litellm.completion_cost(completion_response=response)
94            per_item_cost = cost / len(batch)
95            for item in response.data:
96                results.append(
97                    EmbeddingResult(
98                        embedding=item["embedding"],
99                        model=response.model,
100                        provider=model.split("/")[0],
101                        token_count=response.usage.total_tokens // len(batch),
102                        cost_usd=per_item_cost,
103                    )
104                )
105        return results

• select_model prices candidate models against litellm.model_cost and returns the cheapest within budget, so bulk ingestion stays on the lowest-cost provider automatically.
• embed builds a failover chain starting from the routed choice, retries each model up to max_retries times with a small backoff, and only raises after every model in the chain is exhausted.
• embed_batch chunks the input into batch_size-sized groups, submits each as a single API call, and amortizes the call cost evenly across items so per-chunk cost tracking stays meaningful.

You'll know it works when a single EmbeddingGateway(models=["openai/text-embedding-3-small", "cohere/embed-english-v4.0"]) instance can embed both individual strings and lists of thousands, transparently switching providers when one returns a 429 or 5xx.

Do's and Don'ts

Having walked through the gateway code, the following rules distill the practices that keep a LiteLLM-routed embedding pipeline cheap, observable, and easy to migrate.

Do's

1. ✓Do route every embedding call through the gateway — even one direct provider import becomes the migration that blocks the next vendor swap.
2. ✓Do price requests with litellm.model_cost before dispatch — pre-flight token counting catches budget violations cheaper than an end-of-month invoice.
3. ✓Do order the failover chain by both cost and reliability — cheapest-first is a good default, but pin a known-stable model at the end of the chain.

Don'ts

1. ✗Don't embed one chunk at a time in production loops — per-item calls multiply latency and overhead with no upside over batched aembedding.
2. ✗Don't swallow exceptions inside the failover loop — log model, attempt, and error so degraded providers stay observable.
3. ✗Don't hard-code provider SDKs alongside LiteLLM — mixing call paths defeats the abstraction you just paid for.