Deploy Redis Semantic Cache and Measure Hit Rate vs Cost Savings

Introduction

When you build GenAI applications that handle repeated or paraphrased user queries, sending every prompt to the LLM wastes money and adds latency — even when users are asking semantically identical questions in different words. A semantic cache solves this by matching incoming prompts against stored responses using embedding similarity rather than exact string comparison, turning duplicate intent into cache hits regardless of phrasing. By the end of this lesson, you'll be able to deploy a Redis-backed semantic cache, configure a cosine similarity threshold, and instrument hit rate and cost savings metrics using Prometheus counters.

Key Terminology

• Semantic Cache — A caching layer that stores LLM responses keyed by embedding vectors rather than exact prompt strings, so incoming queries are matched by meaning and paraphrased prompts return cached results without issuing a new LLM call.
• Embedding Vector — A high-dimensional numerical representation of a text prompt produced by calling text-embedding-3-small, yielding a 1536-dimensional vector; each CacheEntry stores this vector alongside the original response so future queries can be compared against it during lookup.
• Cosine Similarity — A measure of angular closeness between two embedding vectors, computed as the dot product of the vectors divided by the product of their magnitudes; the _cosine_similarity method uses it to quantify how semantically alike an incoming prompt is to each cached entry.
• Similarity Threshold — The minimum cosine similarity score (similarity_threshold, defaulting to 0.95) that a candidate cache entry must meet for lookup to return a hit; entries scoring below it are treated as misses so semantically distinct prompts never share a cached response.
• Hit Rate — The fraction of incoming lookups that return a cached response instead of reaching the LLM, tracked in real time by the semantic_cache_hit_rate Prometheus Gauge and derived from the ratio of semantic_cache_hits_total to total lookups labeled by model.
• Prometheus Instrumentation — The set of Counter, Histogram, and Gauge objects registered in the SemanticCache constructor — including semantic_cache_hits_total, semantic_cache_misses_total, semantic_cache_similarity, semantic_cache_savings_dollars, and semantic_cache_hit_rate — that make cache economics visible and comparable across models at runtime.

Concepts

Why Exact-Match Caching Fails for LLM Applications

Traditional caches key responses to an exact input string. That model breaks immediately for natural language: "Explain photosynthesis" and "How does photosynthesis work?" are semantically identical requests, but they produce different cache keys and both reach the LLM — doubling cost and latency for no benefit. Users rephrase, auto-correct, and vary formality constantly, so an exact-match layer achieves close to zero hit rate on real conversational traffic.

A semantic cache replaces the equality check with a similarity check. Each stored entry carries an embedding vector representing the original prompt's meaning, and every incoming query is also vectorized before lookup. If a stored vector is similar enough to the query vector, the cached response is returned directly — regardless of surface-level wording differences (see Code Walkthrough).

Embedding Vectors and Cosine Similarity as a Matching Signal

An embedding model maps text to a point in a high-dimensional vector space — here, text-embedding-3-small produces 1536-dimensional vectors. Prompts that share intent cluster geometrically close together; prompts with different intent land far apart. Cosine similarity measures the angle between two such vectors: a score of 1.0 means perfectly aligned, 0.0 means orthogonal. Because the vector magnitudes are divided out, cosine similarity is insensitive to prompt length, making it a stable matching signal for queries that differ only in verbosity or phrasing.

During lookup, the method computes cosine similarity between the incoming query vector and every embedding stored under the Redis namespace. All scores are observed into the semantic_cache_similarity histogram, giving you a live distribution of how semantically diverse your traffic is — information you need before tuning the threshold (see Code Walkthrough).

The Threshold Trade-off: Precision vs. Correctness

The similarity_threshold parameter is the decision boundary that separates a hit from a miss. Set it too high and you reject valid paraphrases, leaving hit rate — and cost savings — on the table. Set it too low and semantically distinct prompts share a cache entry, returning factually wrong responses to users. That is a correctness failure, not merely a cache-efficiency problem.

The default of 0.95 is deliberately conservative: it captures close paraphrases while rejecting tangentially related queries. The similarity histogram makes this trade-off empirical rather than guesswork — if your traffic distribution shows most near-duplicates clustering between 0.92 and 0.95, you can safely lower the threshold to 0.92 and capture more hits. For high-stakes domains where a wrong cached answer causes real harm, raising toward 0.98 is the appropriate adjustment.

Instrumenting Cache Economics with Prometheus

Cache ROI is invisible without measurement. The SemanticCache constructor registers five Prometheus instruments: hit and miss counters labeled by model (so gpt-4o and claude-3-5-sonnet are compared side by side), a similarity histogram with buckets clustered near the default threshold, a dollar-savings counter, and a real-time hit-rate gauge. Every cache hit increments semantic_cache_hits_total and can credit semantic_cache_savings_dollars with the avoided inference cost — making the cumulative return on the Redis infrastructure directly queryable from your metrics stack.

This instrumentation is what connects the semantic cache to the chapter's broader cost-benefit analysis: without counters and gauges flowing into Prometheus, you can deploy a working semantic cache and still have no evidence that it is paying for itself.

Code Walkthrough

Now that you understand semantic similarity thresholds, embedding vectors, and the hit/miss economics of a cache layer, you can see how those concepts map directly to running Redis code.

The SemanticCache class below wires together every piece: a Redis connection for storage, an OpenAI embeddings call to vectorize prompts, cosine similarity matching to decide whether an incoming prompt is "close enough" to a cached one, and Prometheus metrics to track hit rate and accumulated dollar savings in real time.

Code snippetpython
1import json
2import time
3import numpy as np
4import redis
5from dataclasses import dataclass
6from typing import Optional
7from openai import OpenAI
8from prometheus_client import Counter, Histogram, Gauge
9
10@dataclass
11class CacheEntry:
12    prompt: str
13    response: str
14    embedding: list[float]
15    model: str
16    created_at: float
17    ttl_seconds: int = 3600
18
19@dataclass
20class CacheResult:
21    hit: bool
22    response: Optional[str] = None
23    similarity: float = 0.0
24    latency_ms: float = 0.0
25    saved_cost: float = 0.0
26
27class SemanticCache:
28    def __init__(
29        self,
30        redis_url: str = "redis://localhost:6379",
31        similarity_threshold: float = 0.95,
32        default_ttl: int = 3600,
33        namespace: str = "semcache",
34    ):
35        self.redis = redis.from_url(redis_url)
36        self.threshold = similarity_threshold
37        self.default_ttl = default_ttl
38        self.ns = namespace
39        self.openai = OpenAI()
40        self.hits = Counter("semantic_cache_hits_total", "Cache hits", ["model"])
41        self.misses = Counter("semantic_cache_misses_total", "Cache misses", ["model"])
42        self.similarity_hist = Histogram(
43            "semantic_cache_similarity",
44            "Similarity scores for lookups",
45            buckets=[0.80, 0.85, 0.90, 0.92, 0.95, 0.98, 1.0],
46        )
47        self.savings_counter = Counter(
48            "semantic_cache_savings_dollars",
49            "Cumulative dollar savings from hits",
50            ["model"],
51        )
52        self.hit_rate_gauge = Gauge(
53            "semantic_cache_hit_rate", "Current hit rate", ["model"]
54        )
55
56    def _embed(self, text: str) -> list[float]:
57        resp = self.openai.embeddings.create(
58            model="text-embedding-3-small", input=text
59        )
60        return resp.data[0].embedding
61
62    def _cosine_similarity(self, a: list[float], b: list[float]) -> float:
63        va, vb = np.array(a), np.array(b)
64        return float(np.dot(va, vb) / (np.linalg.norm(va) * np.linalg.norm(vb)))
65
66    def lookup(self, prompt: str, model: str = "gpt-4o") -> CacheResult:
67        start = time.monotonic()
68        query_vec = self._embed(prompt)
69        best_sim, best_entry = 0.0, None
70
71        for key in self.redis.scan_iter(f"{self.ns}:*"):
72            raw = self.redis.get(key)
73            if raw is None:
74                continue
75            entry = CacheEntry(**json.loads(raw))
76            sim = self._cosine_similarity(query_vec, entry.embedding)
77            self.similarity_hist.observe(sim)
78            if sim > best_sim:
79                best_sim, best_entry = sim, entry
80
81        latency_ms = (time.monotonic() - start) * 1000
82        if best_entry and best_sim >= self.threshold:
83            self.hits.labels(model=model).inc()
84            return CacheResult(
85                hit=True,
86                response=best_entry.response,
87                similarity=best_sim,
88                latency_ms=latency_ms,
89            )
90        self.misses.labels(model=model).inc()
91        return CacheResult(hit=False, similarity=best_sim, latency_ms=latency_ms)

The constructor establishes the Redis connection and registers five Prometheus instruments: hit and miss counters (labelled by model so you can compare cache effectiveness across gpt-4o versus claude-3-5-sonnet), a similarity histogram with buckets clustered near the 0.95 default threshold, a dollar-savings counter, and a real-time hit-rate gauge. The _embed helper calls text-embedding-3-small, which produces 1536-dimensional vectors at a fraction of a full inference cost. The lookup method scans all namespace keys, computes cosine similarity against each stored embedding, and returns the best match only when it clears the configured threshold — keeping false positives out of the cache at the expense of a linear scan that you replace with an approximate nearest-neighbor index once the entry count grows past a few thousand.

With the class deployed and a Redis instance running, trigger a few lookups and then query Prometheus to confirm the metrics are flowing:

Code snippetpython
1cache = SemanticCache(redis_url="redis://localhost:6379", similarity_threshold=0.95)
2
3# First call — cold miss, populates nothing yet (illustrates the lookup path)
4result_a = cache.lookup("Explain photosynthesis", model="gpt-4o")
5print(f"Hit: {result_a.hit}, similarity: {result_a.similarity:.3f}")
6
7# After storing a response, a paraphrased prompt should hit above threshold
8result_b = cache.lookup("How does photosynthesis work?", model="gpt-4o")
9print(f"Hit: {result_b.hit}, similarity: {result_b.similarity:.3f}, "
10      f"latency: {result_b.latency_ms:.1f} ms")

Confirm that result_b.hit is True and result_b.similarity is at or above 0.95 — you'll know it works when the second query returns a cached response and the semantic_cache_hits_total Prometheus counter increments by one without a new LLM call being issued.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do tune similarity_threshold using the similarity histogram's bucket data before going to production — The SemanticCache constructor registers a Prometheus histogram with buckets clustered at 0.80, 0.85, 0.90, 0.92, 0.95, 0.98, and 1.0 precisely so you can inspect your domain's actual similarity distribution and decide whether the default 0.95 threshold is too strict (missing valid paraphrases) or too loose (returning wrong cached answers).
2. ✓Do label every Prometheus instrument by model so you can measure cache ROI per model separately — hits, misses, and savings_counter all carry a model label; without it, a high-volume call to a cheaper model inflates the apparent hit rate and hides whether gpt-4o or claude-3-5-sonnet responses are actually being reused, making cost-savings claims unauditable.
3. ✓Do plan to replace redis.scan_iter with an approximate nearest-neighbor index before entry counts grow past a few thousand — lookup() computes _cosine_similarity against every key under the namespace on every query; the O(n) linear scan is intentional at small scale but erases the latency benefit the cache is designed to deliver once the stored entry count grows large enough to make each scan expensive.

Don'ts

1. ✗Don't share a Redis namespace across models that use different embedding spaces — lookup() calls redis.scan_iter(f"{self.ns}:*") and computes cosine similarity against every stored entry indiscriminately; text-embedding-3-small vectors occupy a different geometric space than embeddings from any other model, so cross-model comparisons produce spuriously high similarity scores and cause the cache to return wrong cached responses.
2. ✗Don't skip verifying that result_b.hit is True and result_b.similarity >= 0.95 after storing an entry — If semantic_cache_hits_total never increments and result_b.hit stays False, the cache is silently issuing a full LLM call on every query; the most common culprits are a similarity_threshold set higher than the stored embedding's actual cosine distance or a namespace mismatch that leaves scan_iter finding no keys to compare against.
3. ✗Don't instrument only hits and misses and omit savings_counter and hit_rate_gauge — Hit count alone cannot prove the cache is economically worthwhile; semantic_cache_savings_dollars accumulates per-model dollar savings so you can compare cache profit against the text-embedding-3-small embedding cost incurred on every lookup, and semantic_cache_hit_rate surfaces real-time efficiency decay as query diversity shifts — both are necessary for the cost-benefit analysis this chapter is built around.