Design cache-friendly prompt architectures

Introduction

Every call to an LLM API incurs token costs and latency — but repeated calls that share the same instructions, domain context, and examples can reuse a cached prefix, slashing both. The problem is that prefix caching breaks silently the moment any dynamic value (a timestamp, a user ID, a shuffled example list) appears before the end of the shared block, and most codebases have no structural guard against this. After completing this lesson, you will be able to design prompt template classes that enforce the static-before-dynamic ordering rule, apply cache_control annotations to the correct content blocks, and recognize the anti-patterns that cause cache misses even when caching is nominally enabled.

Key Terminology

• Prefix caching — a provider-side optimization where the token computations for a repeated prompt prefix are stored and reused across successive API calls, reducing both cost and latency for any content that appears in the cached region.
• Static-before-dynamic ordering — the structural rule that all invariant content (system instructions, domain knowledge, few-shot examples) must be assembled before any request-specific values, because prefix caching matches the prompt byte-for-byte from the beginning and any dynamic value inserted early invalidates every token that follows it.
• Cache miss — the result of any byte-level difference in the shared prefix, including an embedded timestamp, a reordered example list, or a request-scoped identifier; a miss causes the full prefix to be reprocessed and billed as if no cache existed.
• cache_control annotation — the {"type": "ephemeral"} dict attached to a content block in the system array of an Anthropic API request that signals the provider to cache up to that point in the prompt; dynamic content placed in the messages array is never annotated.
• Prefix hash — a short SHA-256 fingerprint of the assembled static prefix, produced by get_prefix_hash(), used to verify that two requests using the same template instance produce byte-for-byte identical cacheable content before monitoring cache-hit rates in production.
• Static prefix — the concatenated, deterministically ordered block of system_instructions, domain_knowledge, response_format, and few_shot_examples that CacheAwarePromptTemplate.get_static_prefix() assembles once and places under cache_control; nothing request-specific ever enters this region.

Concepts

Why Prefix Caching Breaks Silently

Prefix caching at providers like Anthropic operates on exact byte-level equality: the provider hashes the incoming token stream from the beginning and considers a cache hit only when every byte up to the annotated boundary is identical to a previously cached call. There is no fuzzy matching and no partial credit. This means a single character of variation — a timestamp printed into the system prompt, a user ID embedded in the instructions, or even a different random seed that reorders few-shot examples — causes a complete cache miss and full reprocessing of the entire prefix. The dangerous part is that this failure is silent: the API still returns correct completions, costs and latency silently revert to uncached levels, and there is no error to alert the developer that caching is not working.

The Static-Before-Dynamic Structural Rule

The only reliable defense is a structural one: make it architecturally impossible for dynamic content to appear before the end of the cacheable region. The CacheAwarePromptTemplate class in this lesson encodes that rule as class design. System instructions, domain knowledge, response format, and few-shot examples are stored as separate, immutable dataclass fields. The method get_static_prefix() assembles them in a fixed, deterministic order — never interpolating request-time values. Dynamic inputs such as the user query and optional per-request context are kept entirely in the messages array, well past the cache_control boundary (see Code Walkthrough). This approach means the ordering rule is enforced by the compiler, not by developer discipline or code-review discipline alone.

Annotating Prompts for Caching

The Anthropic API expects the cacheable region to be declared explicitly. The build_for_anthropic() method wraps the assembled static prefix in a content block that includes "cache_control": {"type": "ephemeral"}, placed in the system array. Dynamic content — the user query and any request-scoped context — is placed in the messages array with no annotation. This separation reflects the mental model: the system array is the stable contract (instructions and context that do not change between calls), and the messages array is the conversation (everything that varies). Annotating a block that actually changes between requests does not cause an error; it just causes a miss every time, wasting the annotation.

Verifying Cache Consistency Before Production

Before relying on cache savings, it is worth confirming that the static prefix is in fact stable. The get_prefix_hash() method computes a 16-character hex fingerprint of whatever get_static_prefix() returns. Calling it twice on the same template instance — even across requests — must yield the same string. If the hashes diverge, dynamic content has leaked into the static region. This check is cheap to add to a staging smoke test and catches accidental mutations (e.g., a few_shot_examples list being appended to in-place) long before they silently erode cache hit rates in production.

Code Walkthrough

Now that you've seen why prefix caching breaks silently, the static-before-dynamic structural rule, annotating prompts for caching, and verifying cache consistency before production, this walkthrough turns them into working code.

The most critical principle in cache-friendly design is placing all static content before any dynamic content. Prefix caching operates on exact byte-level matches — any dynamic value embedded in a static system prompt (a timestamp, a session ID, a shuffled example list) causes a complete cache miss for every token that follows it.

The template below enforces static-before-dynamic ordering at the class level. Static components — system instructions, domain knowledge, response format, and few-shot examples — are stored as separate attributes and assembled once, making it structurally impossible to accidentally inject dynamic content into the cacheable prefix:

Code snippetpython
1from dataclasses import dataclass, field
2from typing import List, Dict, Any, Optional
3from abc import ABC, abstractmethod
4import hashlib
5
6class PromptTemplate(ABC):
7    """
8    Abstract base class for cache-optimized prompt templates.
9
10    Templates enforce the static-before-dynamic pattern
11    that maximizes cache hit rates.
12    """
13
14    @abstractmethod
15    def get_static_prefix(self) -> str:
16        """Return the cacheable static content."""
17        pass
18
19    @abstractmethod
20    def build_prompt(self, dynamic_content: str) -> str:
21        """Build complete prompt with dynamic content at end."""
22        pass
23
24@dataclass
25class CacheAwarePromptTemplate:
26    """
27    Template enforcing cache-friendly prompt structure.
28
29    Guarantees that static content always appears before dynamic
30    content, maximizing cache efficiency.
31
32    Attributes:
33        system_instructions: Core behavioral instructions
34        domain_knowledge: Static domain information
35        response_format: Output format guidelines
36        few_shot_examples: Example interactions
37    """
38    system_instructions: str
39    domain_knowledge: str = ""
40    response_format: str = ""
41    few_shot_examples: List[Dict[str, str]] = field(default_factory=list)

The instance methods complete the design: get_static_prefix() assembles components in deterministic order, get_prefix_hash() generates a fingerprint for cache-hit monitoring, and build_for_anthropic() places the assembled prefix in the system array with a cache_control annotation while keeping dynamic user content in the messages array where it belongs:

Code snippetpython
1    def get_static_prefix(self) -> str:
2        parts = [self.system_instructions]
3
4        if self.domain_knowledge:
5            parts.append(f"\n\nDomain Knowledge:\n{self.domain_knowledge}")
6
7        if self.response_format:
8            parts.append(f"\n\nResponse Format:\n{self.response_format}")
9
10        if self.few_shot_examples:
11            examples_text = "\n\nExamples:\n"
12            for i, example in enumerate(self.few_shot_examples, 1):
13                examples_text += f"\nExample {i}:\n"
14                examples_text += f"User: {example.get('user', '')}\n"
15                examples_text += f"Assistant: {example.get('assistant', '')}\n"
16            parts.append(examples_text)
17
18        return "".join(parts)
19
20    def get_prefix_hash(self) -> str:
21        prefix = self.get_static_prefix()
22        return hashlib.sha256(prefix.encode()).hexdigest()[:16]
23
24    def build_for_anthropic(
25        self,
26        user_query: str,
27        context: Optional[str] = None
28    ) -> Dict[str, Any]:
29        system_blocks = [{
30            "type": "text",
31            "text": self.get_static_prefix(),
32            "cache_control": {"type": "ephemeral"}
33        }]
34
35        user_content = user_query
36        if context:
37            user_content = f"Context:\n{context}\n\nQuestion: {user_query}"
38
39        return {
40            "system": system_blocks,
41            "messages": [{"role": "user", "content": user_content}]
42        }

The cache_control: {"type": "ephemeral"} annotation on the system block signals the provider to cache the prefix; dynamic content such as the user query and optional context lives in the messages array and is never annotated for caching. Note that even a single character of variation — an embedded timestamp, a reordered example, a request-specific identifier — causes a complete cache miss, because providers match prefixes at the byte level.

To verify the template is working correctly, call template.get_prefix_hash() on two separate requests using the same template instance — both calls must return an identical 16-character hex string, confirming that the static prefix is byte-for-byte consistent between requests.

Do's and Don'ts

Having walked through designing cache-friendly prompt architectures above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do store system_instructions, domain_knowledge, response_format, and few_shot_examples as separate dataclass attributes and assemble them through get_static_prefix() — this structural separation makes it impossible to accidentally inject a dynamic value (a timestamp, a session ID) into the cacheable prefix at construction time, which is the most common source of silent cache misses.
2. ✓Do place the assembled static prefix in the system array with cache_control: {"type": "ephemeral"} and route the user query and any per-request context exclusively through the messages array in build_for_anthropic() — annotating the system block and keeping dynamic content out of it is what signals the provider to cache the prefix; mixing dynamic values into that block forces the provider to treat the whole block as uncacheable.
3. ✓Do call template.get_prefix_hash() on successive requests that share the same template instance and verify that both calls return the identical 16-character hex string — a hash mismatch is the earliest detectable signal that a dynamic value has leaked into the static prefix and will produce a byte-level cache miss on every call, making the cache_control annotation effectively inert.

Don'ts

1. ✗Don't embed timestamps, request-scoped identifiers, or reordered few-shot example lists anywhere inside the text assembled by get_static_prefix() — prefix caching matches at the exact byte level, so even a single character of variation in the shared block causes a complete cache miss for every token that follows it, eliminating the cost and latency benefit entirely without any visible error.
2. ✗Don't build prompts by ad-hoc string concatenation at call time instead of using a CacheAwarePromptTemplate instance — without the class-level separation of static attributes from runtime arguments, there is no structural guard against prepending dynamic content to the cacheable portion, and such misses are silent: the API call succeeds but cache_control never produces a hit.
3. ✗Don't apply cache_control: {"type": "ephemeral"} to the messages array entries that carry the user query or per-request context — dynamic content annotated for caching wastes a cache slot on content that will never produce a hit (the payload changes every request), and it signals a misunderstanding of the static-before-dynamic contract that build_for_anthropic() enforces.