Build semantic caching using Redis LangCache

Introduction

When your retrieval pipeline runs the full embed → search → rerank → generate loop for every user question, you pay LLM and vector-DB compute for queries you already answered minutes ago. Teams that ship RAG to production discover that a large fraction of traffic is semantic duplicates — "reset my password" and "how do I reset my password?" hit the same chunks and produce the same answer, but cost the same dollars. Without a cache that matches on meaning rather than exact text, you burn budget on work you've already done, and tail latency climbs as load grows. By the end of this lesson you'll be able to build a Redis-backed semantic cache that compares query embeddings against stored ones, returns hits above a tunable similarity threshold, and lets you sweep that threshold against an evaluation set to pick the right accuracy/savings tradeoff.

Key Terminology

• Semantic cache — a cache keyed by meaning (a query embedding) rather than literal text, so paraphrased queries that mean the same thing share a single cached answer.
• Cosine similarity — the dot product of two L2-normalized vectors, bounded in [-1, 1]; the score the cache uses to decide whether an incoming embedding is "close enough" to a stored one.
• Similarity threshold — the cosine-similarity cutoff above which a cached entry counts as a hit; the single knob that trades hit rate against the risk of returning a wrong answer.
• Cache-aside pattern — the application checks the cache first, falls through to the full pipeline on a miss, and writes the result back; the cache never sits in the request path on its own.
• TTL (time-to-live) — the lifetime Redis enforces on each cached entry, which bounds staleness when the underlying corpus or embedding model changes.

Concepts

Embedding-based lookup vs. exact-key lookup

A traditional key-value cache returns a hit only when the incoming key matches a stored key byte-for-byte. That fails the moment a user paraphrases — "reset password" and "how do I reset my password?" produce different keys and miss. A semantic cache replaces equality with a similarity score: embed the incoming query, compare its embedding against every stored query embedding using cosine similarity, and treat anything above the threshold as a hit. The cached result is what gets returned; the cached query embedding only exists so similarity can be computed (see Code Walkthrough).

Threshold as an accuracy/savings dial

The similarity threshold is the only runtime knob that matters. Set it too low (e.g. 0.80) and you serve cached answers to queries that look similar but aren't — "reset password" vs "reset PIN" will collide. Set it too high (e.g. 0.98) and almost nothing matches, and the savings collapse. Production deployments commonly land between 0.90 and 0.94, but the right value is the one your evaluation set tells you — never a hard-coded default. The threshold sweep later in this lesson is how you produce that evidence.

Why Redis as the backing store

Redis gives the semantic cache three properties it needs: sub-millisecond reads (so the lookup is cheaper than the LLM call it replaces), native TTL enforcement (entries expire automatically when the corpus or model rotates), and RediSearch vector indexes for production-scale similarity search. The walkthrough below uses a linear scan over KEYS qcache:* for clarity — once you cross roughly 10k entries, switch to a RediSearch HNSW index for O(log n) lookup.

Code Walkthrough

The two snippets below implement the two concepts above: SemanticQueryCache does the embedding-based lookup and Redis-backed storage under the cache-aside pattern, and ThresholdTuner sweeps thresholds against an eval set so the threshold you ship is evidence-backed, not guessed.

Code snippetpython
1import redis.asyncio as redis
2import json
3import hashlib
4import numpy as np
5
6class SemanticQueryCache:
7    def __init__(
8        self,
9        redis_url: str = "redis://localhost:6379",
10        similarity_threshold: float = 0.92,
11        ttl_seconds: int = 3600,
12    ):
13        self.client = redis.from_url(redis_url)
14        self.threshold = similarity_threshold
15        self.ttl = ttl_seconds
16
17    async def get_cached(
18        self,
19        query_embedding: list[float],
20    ) -> dict | None:
21        keys = await self.client.keys("qcache:*")
22        best_match = None
23        best_sim = 0.0
24        query_arr = np.array(query_embedding)
25        query_norm = query_arr / np.linalg.norm(query_arr)
26        for key in keys:
27            cached = await self.client.get(key)
28            if cached is None:
29                continue
30            data = json.loads(cached)
31            cached_arr = np.array(data["query_embedding"])
32            cached_norm = cached_arr / np.linalg.norm(cached_arr)
33            sim = float(np.dot(query_norm, cached_norm))
34            if sim > self.threshold and sim > best_sim:
35                best_sim = sim
36                best_match = data
37        if best_match:
38            return {
39                "results": best_match["results"],
40                "similarity": best_sim,
41                "cached_query": best_match["query_text"],
42            }
43        return None
44
45    async def store(
46        self,
47        query_text: str,
48        query_embedding: list[float],
49        results: list[dict],
50    ) -> None:
51        key_hash = hashlib.sha256(query_text.encode()).hexdigest()[:16]
52        key = f"qcache:{key_hash}"
53        payload = json.dumps({
54            "query_text": query_text,
55            "query_embedding": query_embedding,
56            "results": results,
57        })
58        await self.client.setex(key, self.ttl, payload)

get_cached L2-normalizes the incoming embedding once, then walks every stored entry, computes cosine similarity as np.dot of the normalized vectors, and keeps the best match above the threshold. store keys each entry by a SHA-256 of the query text (so identical text overwrites instead of duplicating) and lets Redis enforce the TTL via SETEX. The 0.92 default is only a starting point — replace it with the value the next snippet's sweep recommends.

Code snippetpython
1class ThresholdTuner:
2    def __init__(self, eval_queries: list[dict]):
3        self.queries = eval_queries
4
5    def evaluate_threshold(
6        self,
7        threshold: float,
8        query_embeddings: list[list[float]],
9        gold_results: list[list[dict]],
10    ) -> dict:
11        hits = 0
12        correct_hits = 0
13        total = len(self.queries)
14        for i in range(total):
15            for j in range(total):
16                if i == j:
17                    continue
18                sim = float(np.dot(
19                    np.array(query_embeddings[i]),
20                    np.array(query_embeddings[j]),
21                ))
22                if sim >= threshold:
23                    hits += 1
24                    if gold_results[i] == gold_results[j]:
25                        correct_hits += 1
26                    break
27        hit_rate = hits / max(total, 1)
28        accuracy = correct_hits / max(hits, 1)
29        return {
30            "threshold": threshold,
31            "hit_rate": round(hit_rate, 4),
32            "accuracy": round(accuracy, 4),
33            "estimated_savings": round(hit_rate * accuracy, 4),
34        }
35
36    def sweep(
37        self,
38        thresholds: list[float],
39        query_embeddings: list[list[float]],
40        gold_results: list[list[dict]],
41    ) -> list[dict]:
42        return [
43            self.evaluate_threshold(t, query_embeddings, gold_results)
44            for t in thresholds
45        ]

evaluate_threshold simulates cache behaviour on an eval set: for each query, it asks "is there another query whose embedding sits above the threshold, and would that other query's gold result still be acceptable for me?" A "correct hit" requires both conditions. sweep runs that evaluation across a list of thresholds — typically [0.85, 0.88, 0.90, 0.92, 0.94, 0.96] — so you can watch hit rate and accuracy move in opposite directions. You'll know it works when the sweep produces a monotone-decreasing hit_rate and a monotone-increasing accuracy as the threshold rises, and the estimated_savings column has a single clear peak — that peak is the threshold to deploy.

Do's and Don'ts

Having just walked through the cache implementation and the threshold sweep, the rules below are the operational guardrails that keep semantic caching honest in production.

Do's

1. ✓Do pick the threshold from a sweep, not a default — production behaviour varies by domain and embedding model; the right value is the one that maximizes hit_rate * accuracy on your eval set.
2. ✓Do set a TTL shorter than your corpus or model rotation cadence — Redis evicts stale entries for you, so you never serve answers built from an embedding that no longer matches how the current model represents the corpus.
3. ✓Do switch to a RediSearch HNSW index once you cross ~10k cached entries — the linear KEYS qcache:* scan is fine for prototypes but becomes the cache's bottleneck before you notice.

Don'ts

1. ✗Don't cache the assembled prompt + context as the key — the matching surface should be the query embedding, not the full prompt bundle, otherwise paraphrases never hit.
2. ✗Don't skip L2 normalization before np.dot — without it you're computing raw dot products instead of cosine similarity, and threshold values stop being comparable across runs or embedding models.
3. ✗Don't trust a high hit rate alone — 80% hits at 60% accuracy is worse than 50% hits at 95% accuracy; always read both columns of the sweep together.