Build a semantic cache with Redis + embedding similarity

Introduction

When you're paying per token on every LLM call, you'll notice that users keep asking the same thing in slightly different words — "Summarize Q3 earnings" and "Give me a third-quarter recap" hit your provider as two billable requests even though one cached answer would serve both. Skip semantic caching and you keep paying for duplicate work, watching latency and spend climb in lockstep with traffic. By the end of this lesson, you'll be able to build a Redis-backed semantic cache that embeds incoming queries, matches them against stored entries by cosine similarity, and serves cached responses whenever similarity clears your threshold.

Key Terminology

• Semantic cache: A cache keyed on the meaning of a query rather than its exact text, so paraphrased requests that mean the same thing resolve to one stored response.
• Query embedding: A fixed-length vector produced by an embedding model that encodes a query's meaning, enabling similarity comparison between differently worded requests.
• Cosine similarity: The dot product of two normalized embedding vectors, scoring how close two queries are in meaning on a 0–1 scale; it is the match metric the cache thresholds on.

Concepts

How a Redis-backed semantic cache decides what counts as a match and how it scales.

Why Semantic Caching Matters for Cost Control

Before diving into implementation, consider the economics. A single GPT-4o request processing 1,000 input tokens and generating 500 output tokens costs roughly $0.00875. An embedding call to text-embedding-3-small for the same 1,000-token query costs approximately $0.00002—over 400× cheaper. If even 20% of your traffic matches a cached response semantically, the savings compound rapidly. At 100,000 daily requests, a 20% semantic hit rate eliminates approximately 20,000 LLM calls per day, saving over $150 daily on GPT-4o alone. The embedding overhead for all 100,000 lookups adds roughly $2.00. This asymmetry is what makes semantic caching one of the highest-ROI optimizations in the cost-tracking stack, complementing the prompt caching strategies covered earlier for Anthropic and OpenAI and feeding directly into the budget enforcement layer that throttles spending.

• Semantic similarity threshold: The cosine similarity score (typically 0.92–0.98) above which two queries are considered equivalent. Too low produces false positives; too high collapses to near-exact matching.
• Embedding model: A lightweight model like text-embedding-3-small (1536 dimensions) optimized for speed and cost rather than the larger text-embedding-3-large (3072 dimensions).
• Cache TTL: Time-to-live controlling staleness. Factual queries tolerate longer TTLs (hours); rapidly changing data demands shorter windows (minutes).
• Candidate set: The subset of cached entries compared against the incoming query. Brute-force comparison works below ~50,000 entries; beyond that, approximate nearest-neighbor indices (FAISS, Redis VSS) become necessary.

Tuning the Similarity Threshold

The similarity threshold is the single most impactful configuration parameter. Setting it too low (e.g., 0.85) creates false positives where semantically distinct queries return incorrect cached responses—a user asking "What are the risks of LLM hallucination?" might match against "What are the benefits of LLM hallucination?" at 0.88 similarity, returning a completely wrong answer. Setting it too high (e.g., 0.99) collapses the cache to near-exact matching, eliminating the advantage over simple string hashing.

Production tuning follows a three-step process. First, collect a sample of 500–1,000 real query pairs from your application logs and manually label them as semantically equivalent or distinct. Second, compute pairwise cosine similarities using your chosen embedding model. Third, plot the distribution and select a threshold that maximizes true positive hits while keeping false positives below 1%. In practice, text-embedding-3-small with a threshold of 0.95 achieves this balance for most English-language GenAI applications. Domain-specific terminology (medical, legal, financial) often requires raising the threshold to 0.96–0.97 because specialized terms carry more semantic weight per token.

Scaling Beyond Brute-Force: Redis Vector Search

The SCAN-based candidate iteration shown above works reliably for caches containing up to approximately 50,000 entries. Beyond that scale, linear comparison becomes a latency bottleneck. Redis Stack provides native vector similarity search (VSS) via the FT.SEARCH command with VECTOR fields, supporting both flat (brute-force) and HNSW (approximate nearest-neighbor) indexing. Migrating from the SCAN approach to Redis VSS requires three changes: creating a vector index on your namespace with FT.CREATE, storing embeddings as binary blobs via numpy.array.tobytes() instead of JSON-serialized lists, and replacing the SCAN loop in cache_lookup() with a single FT.SEARCH KNN query. This migration preserves the same SemanticCache interface—only the internal retrieval mechanism changes—so the cached_llm_call() integration layer and cost-tracking events remain untouched.

Measuring Cache Effectiveness

A semantic cache is only worth running if you can prove it is saving more than it costs, so the lookup-and-store path must emit the numbers that make its value measurable. Every CACHE_HIT event carries the avoided LLM cost as a saving, while every lookup—hit or miss—records the embedding cost that the cache always incurs. The two figures define the cache's ROI: net savings are the summed avoided-LLM cost minus the summed embedding cost. The metric that drives both is the hit rate—the fraction of queries that clear the similarity threshold. A 30% hit rate against GPT-4o traffic, where each avoided call dwarfs the ~$0.00002 embedding lookup, turns the cache strongly net-positive; a hit rate near zero means your threshold is too high or your traffic has little semantic overlap, and the embedding overhead is pure loss. Watching hit rate alongside the false-positive rate from threshold tuning is how you confirm the cache is deduplicating real near-duplicates rather than serving wrong answers.

Code Walkthrough

A step-by-step build of the Redis-backed SemanticCache: the query-flow architecture, the lookup-and-store class, and the wrapper that makes caching transparent to the LLM call path.

Architecture: Query Flow Through the Semantic Cache

The following diagram illustrates how an incoming LLM request flows through the semantic cache layer before reaching the provider API. The cache sits between your application's request handler and the LLM client, intercepting queries before they incur token costs. Cache misses proceed to the LLM, and the response is stored alongside its embedding for future lookups. Every interaction—hit or miss—is logged to the cost-tracking pipeline so the budget enforcement engine and analytics views reflect accurate spend and savings data.

Loading diagram...

The critical decision node is the similarity threshold comparison. Every request generates an embedding regardless of cache outcome—this is the fixed cost of semantic caching. The variable cost savings come from avoiding LLM calls on cache hits, which is why the embedding model choice directly impacts ROI.

Core Implementation: The SemanticCache Class

The following implementation defines the SemanticCache class with two primary methods: cache_lookup() and cache_store(). The cache_lookup() method embeds the incoming query using OpenAI's text-embedding-3-small model, retrieves all candidate embeddings from Redis using the SCAN command with a namespace prefix, computes cosine similarity against each candidate using NumPy's dot product, and returns the cached response if any candidate exceeds the configured similarity threshold. The cache_store() method serializes both the embedding vector and the LLM response into a Redis hash with a configurable TTL. The class uses the openai.OpenAI client for embedding generation and redis.Redis for storage, with numpy handling the vector math.

1import json
2import hashlib
3import time
4import numpy as np
5import openai
6import redis
7from dataclasses import dataclass
8
9@dataclass
10class CacheResult:
11    hit: bool
12    response: str | None = None
13    similarity: float = 0.0
14    cached_query: str | None = None
15    embedding_cost: float = 0.0
16
17class SemanticCache:
18    def __init__(
19        self,
20        redis_client: redis.Redis,
21        openai_client: openai.OpenAI,
22        namespace: str = "semcache",
23        similarity_threshold: float = 0.95,
24        default_ttl: int = 3600,
25        embedding_model: str = "text-embedding-3-small",
26    ):
27        self.redis = redis_client
28        self.openai = openai_client
29        self.namespace = namespace
30        self.threshold = similarity_threshold
31        self.ttl = default_ttl
32        self.embedding_model = embedding_model
33
34    def _get_embedding(self, text: str) -> tuple[list[float], int]:
35        resp = self.openai.embeddings.create(
36            input=text, model=self.embedding_model
37        )
38        tokens_used = resp.usage.total_tokens
39        return resp.data[0].embedding, tokens_used
40
41    def _cosine_similarity(self, a: list[float], b: list[float]) -> float:
42        va, vb = np.array(a), np.array(b)
43        denom = np.linalg.norm(va) * np.linalg.norm(vb)
44        if denom == 0:
45            return 0.0
46        return float(np.dot(va, vb) / denom)
47
48    def cache_lookup(self, query: str) -> CacheResult:
49        embedding, tokens = self._get_embedding(query)
50        embed_cost = tokens * 0.00000002  # $0.02 per 1M tokens
51
52        best_sim, best_key = 0.0, None
53        cursor = 0
54        while True:
55            cursor, keys = self.redis.scan(
56                cursor, match=f"{self.namespace}:*", count=100
57            )
58            for key in keys:
59                data = self.redis.hgetall(key)
60                if not data:
61                    continue
62                cached_emb = json.loads(data[b"embedding"])
63                sim = self._cosine_similarity(embedding, cached_emb)
64                if sim > best_sim:
65                    best_sim = sim
66                    best_key = key
67            if cursor == 0:
68                break
69
70        if best_sim >= self.threshold and best_key is not None:
71            data = self.redis.hgetall(best_key)
72            return CacheResult(
73                hit=True,
74                response=data[b"response"].decode(),
75                similarity=best_sim,
76                cached_query=data[b"query"].decode(),
77                embedding_cost=embed_cost,
78            )
79        return CacheResult(hit=False, embedding_cost=embed_cost)
80
81    def cache_store(
82        self, query: str, response: str, embedding: list[float] | None = None
83    ) -> str:
84        if embedding is None:
85            embedding, _ = self._get_embedding(query)
86        key_hash = hashlib.sha256(query.encode()).hexdigest()[:16]
87        cache_key = f"{self.namespace}:{key_hash}"
88        self.redis.hset(cache_key, mapping={
89            "query": query,
90            "response": response,
91            "embedding": json.dumps(embedding),
92            "created_at": str(time.time()),
93        })
94        self.redis.expire(cache_key, self.ttl)
95        return cache_key

Integrating the Cache with the LLM Request Pipeline

The semantic cache must wrap your existing LLM call path so that cache lookups happen transparently. The following function, cached_llm_call(), demonstrates integration with the cost-tracking system. It performs a cache_lookup() first, and on a hit, logs the avoided cost as a saving attributed to semantic caching. On a miss, it forwards the request to the LLM provider, stores the result via cache_store(), and records the actual spend. The log_cost_event() function referenced here feeds into the cost analytics materialized views discussed in the analytics section, ensuring that cache savings appear alongside prompt caching savings from Anthropic and OpenAI in the unified cost dashboard.

1from enum import Enum
2
3class CostSource(str, Enum):
4    LLM_CALL = "llm_call"
5    CACHE_HIT = "semantic_cache_hit"
6    EMBEDDING = "embedding_lookup"
7
8def cached_llm_call(
9    cache: SemanticCache,
10    openai_client: openai.OpenAI,
11    query: str,
12    model: str = "gpt-4o",
13    user_id: str = "",
14    log_cost_event: callable = lambda **kw: None,
15) -> dict:
16    result = cache.cache_lookup(query)
17
18    log_cost_event(
19        user_id=user_id,
20        source=CostSource.EMBEDDING,
21        cost=result.embedding_cost,
22        model=cache.embedding_model,
23    )
24
25    if result.hit:
26        log_cost_event(
27            user_id=user_id,
28            source=CostSource.CACHE_HIT,
29            cost=0.0,
30            savings=_estimate_llm_cost(model, query, result.response),
31            metadata={"similarity": result.similarity},
32        )
33        return {"response": result.response, "cached": True,
34                "similarity": result.similarity}
35
36    completion = openai_client.chat.completions.create(
37        model=model,
38        messages=[{"role": "user", "content": query}],
39    )
40    response_text = completion.choices[0].message.content
41    actual_cost = _compute_cost_from_usage(completion.usage, model)
42
43    cache.cache_store(query, response_text)
44
45    log_cost_event(
46        user_id=user_id,
47        source=CostSource.LLM_CALL,
48        cost=actual_cost,
49        model=model,
50        tokens_in=completion.usage.prompt_tokens,
51        tokens_out=completion.usage.completion_tokens,
52    )
53    return {"response": response_text, "cached": False, "cost": actual_cost}
54
55def _estimate_llm_cost(model: str, query: str, response: str) -> float:
56    import tiktoken
57    enc = tiktoken.encoding_for_model(model)
58    in_tokens = len(enc.encode(query))
59    out_tokens = len(enc.encode(response))
60    rates = {"gpt-4o": (0.005, 0.015), "gpt-4o-mini": (0.00015, 0.0006)}
61    in_rate, out_rate = rates.get(model, (0.005, 0.015))
62    return (in_tokens * in_rate + out_tokens * out_rate) / 1000

Do's and Don'ts

Do's

1. ✓Do tune similarity_threshold deliberately before deploying — the default 0.95 in SemanticCache.__init__ is a starting point, not a universal constant; a threshold that is too low will serve cached answers to semantically different queries ("Q3 earnings" and "Q3 headcount"), while one that is too high collapses the hit rate to near zero and turns every embedding call into pure overhead.
2. ✓Do log every cache hit and cache miss into the shared cost analytics pipeline — the architecture routes both the hit branch (cost savings) and the miss branch (actual API cost) through node K before reaching the budget enforcement check; omitting either branch leaves L blind to real spend and makes the budget gate meaningless.
3. ✓Do use redis.Redis.scan() with a namespace prefix rather than a blocking KEYS command — cache_lookup() iterates with a cursor (cursor, keys = self.redis.scan(cursor, match=f"{self.namespace}:*", count=100)) specifically to avoid blocking the Redis event loop; a KEYS * scan on a large cache keyspace stalls every other Redis client for the full scan duration.

Don'ts

1. ✗Don't ignore the fixed embedding cost when evaluating cache ROI — cache_lookup() calls _get_embedding() on every incoming query regardless of hit or miss, charging tokens * 0.00000002 each time; if your hit rate is low (threshold too high, sparse traffic), the cumulative embedding spend can exceed the LLM call savings, making the semantic cache net-negative.
2. ✗Don't store a Redis cache entry without a TTL — cache_store() is designed to accept a default_ttl (default 3600 seconds); skipping the TTL lets stale embeddings and outdated LLM responses accumulate indefinitely, causing cache_lookup() to serve answers that are no longer correct to semantically similar future queries.
3. ✗Don't conflate cosine similarity with exact-match hashing — _cosine_similarity() computes np.dot(va, vb) / (norm(va) * norm(vb)) over floating-point vectors, not deterministic key equality; two character-identical queries can produce marginally different embeddings across API versions, so building cache keys with hashlib on raw query strings and bypassing the similarity check will silently miss the deduplication that the entire architecture depends on.