Implement model cascading for cost reduction

Introduction

When you serve every RAG query through GPT-4o or Claude Opus, your inference bill scales linearly with traffic — and you are paying frontier prices for queries a fine-tuned 7B model would have answered just as well. Teams that ignore this end up either capping traffic, raising prices, or watching margins collapse as adoption grows. Model cascading routes the easy majority of queries to a cheap model and reserves the expensive model for the hard tail, cutting per-query cost 5-15x while keeping answer quality within 1-2 points of the frontier baseline. By the end of this lesson you will be able to generate distillation data from a frontier model, build a difficulty-aware router, and benchmark the cascade so you can prove the cost savings did not silently erode quality.

Key Terminology

• Knowledge distillation — using outputs from a frontier "teacher" model as supervised training data for a cheaper "student" model, so the student learns to mimic the teacher on a fixed task without ever seeing the teacher's weights. This is how the cheap tier of the cascade becomes good enough to handle real production queries.
• Query difficulty router — the component sitting in front of the model tier that classifies each incoming query as easy or hard based on lightweight features (length, retrieved-chunk count, comparison keywords) and dispatches it to the cheap or expensive model accordingly.
• Quality retention — the percentage of the expensive-model baseline quality that the cascade preserves, measured by an LLM judge over a held-out set. Below ~95% means the cheap tier is leaking quality; above ~98% means the router is probably over-escalating and you are leaving cost savings on the table.
• Cost reduction percentage — the share of inference spend the cascade eliminates versus running everything on the expensive model. Reported per routing tier so easy-tier wins are not masked by hard-tier spend.
• LiteLLM gateway — a thin model-routing layer that exposes a single OpenAI-compatible API in front of many providers (OpenAI, Anthropic, DeepSeek, Together, Groq). Swapping the cheap or expensive model is a string change, not a code change.

Concepts

Knowledge distillation from the frontier model

The cheap model in a cascade is only useful if it can match the frontier model on the subset of queries you route to it. The cheapest reliable way to make that happen is supervised fine-tuning on (query, context, expensive-model-answer) triples drawn from your own traffic. You run the expensive model on a representative sample, score every answer with the LLM judge, keep only the high-scoring examples, and train the cheap model on those. The student never has to generalise far — it just learns the shape of correct answers in your specific domain. See Code Walkthrough for the runtime that consumes the distilled student.

Query difficulty routing

Once the cheap model is trained, the router decides which model each incoming query goes to. The classifier can be a small trained model, but a heuristic over four cheap signals usually gets within 2-3 percentage points of a learned router: query length (longer queries skew harder), number of retrieved context chunks (high counts signal ambiguity), comparison/contrast keywords (multi-facet reasoning), and multi-hop connectives like "and also" or "as well as" (synthesis required). Each signal contributes a partial score; queries above a tunable threshold escalate to the expensive tier. The threshold itself is tuned with the same A/B harness you use for any other production model change.

Cascade benchmarking and the retention/reduction trade-off

A cascade that ships without per-tier metrics is a cascade you cannot trust. The benchmark replays a held-out evaluation set through both the baseline (expensive model on everything) and the cascade, scores both with the same judge, and reports quality retention and cost reduction independently for each tier. Per-tier reporting is what catches a router that is dumping hard queries into the cheap model: an aggregate retention of 98% can hide a hard-tier retention of 80%, and you only see that decomposition when you slice by routing decision.

Code Walkthrough

Now that you have seen the concepts above, the walkthrough below turns them into working code.

The router and the benchmark are the two pieces that make cascading observable — the router decides what runs cheaply, and the benchmark proves the decision was right. The first snippet implements the heuristic router; the second wraps a batch of routed results in a tier-aware benchmark that compares cascade quality and cost against the expensive-only baseline.

Difficulty router

Code snippetpython
1from dataclasses import dataclass
2
3@dataclass
4class RouteDecision:
5    tier: str
6    model: str
7    confidence: float
8    reason: str
9
10def route_query(
11    query: str,
12    context_chunks: list[str],
13    cheap_model: str = "deepseek/deepseek-chat",
14    expensive_model: str = "gpt-4o",
15) -> RouteDecision:
16    query_words = len(query.split())
17    num_chunks = len(context_chunks)
18    q_lower = query.lower()
19    has_comparison = any(
20        w in q_lower
21        for w in ["compare", "versus", "difference", "contrast"]
22    )
23    has_multi_hop = any(
24        w in q_lower
25        for w in ["and also", "in addition", "furthermore", "as well as"]
26    )
27    difficulty = 0.0
28    if query_words > 30:
29        difficulty += 0.3
30    if num_chunks > 5:
31        difficulty += 0.2
32    if has_comparison:
33        difficulty += 0.3
34    if has_multi_hop:
35        difficulty += 0.3
36    if difficulty >= 0.5:
37        return RouteDecision(
38            tier="hard",
39            model=expensive_model,
40            confidence=min(difficulty, 1.0),
41            reason="complex query features",
42        )
43    return RouteDecision(
44        tier="easy",
45        model=cheap_model,
46        confidence=1.0 - difficulty,
47        reason="simple query features",
48    )

• RouteDecision records tier, chosen model, confidence, and a human-readable reason — log the reason so you can audit routing accuracy later without re-running the classifier.
• The four features (length, chunk count, comparison keywords, multi-hop connectives) each contribute a fractional score; the 0.5 threshold is intentionally conservative so borderline queries escalate rather than risk quality regression. Both model identifiers are string-configurable so swapping providers via LiteLLM is a config change.

Per-tier cascade benchmark

Code snippetpython
1from dataclasses import dataclass
2
3@dataclass
4class CascadeBenchmark:
5    tier: str
6    sample_count: int
7    avg_quality_baseline: float
8    avg_quality_cascade: float
9    quality_retention_pct: float
10    cost_reduction_pct: float
11
12def benchmark_cascade(results: list[dict]) -> list[CascadeBenchmark]:
13    tiers: dict[str, list] = {}
14    for r in results:
15        tiers.setdefault(r["tier"], []).append(r)
16    benchmarks = []
17    for tier, items in tiers.items():
18        n = len(items)
19        base_avg = sum(i["baseline_score"] for i in items) / n
20        casc_avg = sum(i["cascade_score"] for i in items) / n
21        base_cost = sum(i["baseline_cost"] for i in items)
22        casc_cost = sum(i["cascade_cost"] for i in items)
23        retention = (casc_avg / base_avg * 100) if base_avg > 0 else 0.0
24        reduction = (1 - casc_cost / base_cost) * 100 if base_cost > 0 else 0.0
25        benchmarks.append(CascadeBenchmark(
26            tier=tier,
27            sample_count=n,
28            avg_quality_baseline=round(base_avg, 3),
29            avg_quality_cascade=round(casc_avg, 3),
30            quality_retention_pct=round(retention, 1),
31            cost_reduction_pct=round(reduction, 1),
32        ))
33    return benchmarks

• Grouping by tier first is what makes the report useful — aggregate-only numbers hide routing failures. If hard-tier retention drops below 95% while easy-tier holds, the router is escalating too little; if easy-tier drops, the cheap model needs more distillation data.
• quality_retention_pct above 95% combined with cost_reduction_pct of 60-85% on the easy tier is the canonical sign the cascade is working; either number alone is not enough.

You'll know it works when a benchmark run over 200+ held-out queries shows easy-tier retention ≥95% and overall cost reduction ≥50% versus the expensive-only baseline.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do report quality retention per routing tier — aggregate retention can mask an 80% hard-tier collapse hiding behind a 98% easy-tier win.
2. ✓Do tune the router threshold with the same A/B harness you use for model swaps — the threshold IS a production knob, not a constant.
3. ✓Do log the routing reason on every request — without it, you cannot diagnose why a query escalated or stayed cheap when the benchmark surprises you.

Don'ts

1. ✗Don't ship a cascade without an LLM-judge benchmark — you have no way to detect quality regression in the cheap tier until users complain.
2. ✗Don't train the cheap model on unfiltered teacher outputs — keep only examples above the judge threshold, or the student inherits the teacher's mistakes.
3. ✗Don't pick the threshold once and freeze it — query distribution drifts, and a router calibrated on last quarter's traffic will silently over- or under-escalate.