Build a context window composer with token budgets

Introduction

When you wire a Claude-powered assistant to long-running conversations, the 200,000-token window fills faster than you expect — a system prompt, twelve tool schemas, thirty user memories, fifty RAG chunks, and two hundred history turns can consume 74,000 tokens before the model emits a single output token. Without explicit per-section budgets, older context is silently evicted, instructions go missing mid-conversation, and the model starts contradicting earlier turns or forgetting user preferences. By the end of this lesson you'll be able to compose a multi-section LLM context window — system prompt, tool schemas, Mem0 memories, RAG chunks, and conversation history — under hard per-section token budgets, with deterministic eviction and Anthropic prompt caching applied to the stable prefix.

Key Terminology

• Context Section: A named region of the message list (system, tools, memories, history) with its own maximum token allocation.
• Token Budget: The upper bound of tokens a section may consume; all section budgets plus the output reserve must sum to no more than the model's context window.
• Stable Prefix: The leading portion of the request (system prompt + tool schemas) that does not change between turns and therefore qualifies for Anthropic prompt caching via cache_control markers.

Concepts

This section explains why a fixed 200,000-token window forces a sectioned, budgeted design, and how to allocate that budget across system prompt, tools, memories, RAG chunks, and conversation history.

Why Section-Based Composition Matters

A Claude 3.5 Sonnet context window holds 200,000 tokens. That sounds generous until you load a system prompt (800 tokens), 12 tool schemas (4,000 tokens), 30 Mem0 memories (1,500 tokens), 50 RAG chunks (8,000 tokens), and 200 conversation turns (60,000 tokens). You have consumed 74,300 tokens before the model generates a single output token—and output tokens count against the same window. Without budgeting, a long-running conversation silently pushes older context out of the window, and the model loses critical instructions or forgets user preferences. Section-based composition solves this by treating the context window as a fixed-capacity container partitioned into named sections, each with an explicit ceiling.

The key terms that underpin this design:

• Context Section: A named region of the message list (e.g., system, tools, memories, history) with a maximum token allocation.
• Token Budget: The upper bound of tokens a section may consume. Budgets across all sections must sum to less than or equal to the model's context window minus a reserved output buffer.
• Eviction Policy: The strategy applied when a section's content exceeds its budget. Conversation history typically uses oldest-first eviction; memories use relevance-score eviction.
• Stable Prefix: The portion of the message list that does not change between requests—system prompt plus tool schemas—which qualifies for Anthropic prompt caching via cache_control ephemeral markers.
• Composition Order: The sequence in which sections appear in the final message list. Anthropic models weight earlier tokens more heavily during attention, so high-priority context (system instructions, tool schemas) occupies the prefix.

Budget Allocation Strategy

Choosing the right token budgets requires understanding your application's access patterns. A customer support bot with short conversations but many tools should allocate more to tools_budget and less to history_budget. A personal assistant with month-long conversations needs the inverse. The following guidelines apply to most production deployments:

1. Reserve 2-4% for output — Claude rarely generates responses longer than 4,096 tokens in conversational settings, but code generation or structured extraction tasks may need 8,192.
2. Fix stable sections first — System prompts and tool schemas have predictable sizes. Measure them once and set budgets with 20% headroom for iteration.
3. Cap memories at 1-2% — Mem0 memories are short factual statements (10-30 tokens each). Twenty memories consume roughly 400-600 tokens. A 2,000-token budget handles even aggressive retrieval.
4. Give history the remainder — Conversation history is the most variable section and benefits most from a large budget. The history_budget property computes this dynamically.
5. Monitor with Redis counts — The per-section token counts persisted to Redis by session_cache.store_counts allow you to build dashboards that reveal when conversations consistently hit eviction thresholds, signaling that budgets need rebalancing.

Code Walkthrough

This walkthrough builds the ContextComposer class end-to-end: first the budget configuration and tokenizer setup, then the assembly pipeline that loads each section, enforces budgets, evicts overflow, and attaches cache_control markers to the stable prefix.

The Composition Pipeline

The following diagram illustrates how the ContextComposer.compose_context method assembles the final message list from four data sources, applies per-section token budgets, and attaches cache_control markers to the stable prefix before returning the payload to the Anthropic API.

ContextComposer.compose_context orchestrates a token-budgeted assembly pipeline that prevents LLM context windows from silently overflowing. It allocates fixed budgets—1,000 tokens for the system prompt, 5,000 for tool schemas, 2,000 for Mem0 memory retrieval—then assigns the remaining window capacity to conversation history. When the assembled payload exceeds the window limit, the eviction loop discards the oldest history turns first. The final step attaches cache_control markers to the stable prefix (system prompt + tool schemas), enabling Anthropic's prompt caching to skip re-processing unchanging context on every call.

Loading diagram...

The critical insight is that the system prompt and tool schemas form a stable prefix—they are identical across every request in a session. By marking the last block of that prefix with {"cache_control": {"type": "ephemeral"}}, Anthropic caches the KV computations for those tokens, reducing input cost by up to 90% on cache hits. Everything after the cache boundary—memories, history, the new user message—varies per request and is computed fresh.

Implementing the ContextComposer

The following implementation defines the ContextComposer class in context/composer.py. This class accepts a BudgetConfig dataclass that specifies per-section token ceilings, a MemoryClient instance from the Mem0 SDK for semantic memory retrieval, an async SQLAlchemy session factory for loading conversation history from PostgreSQL, and a Redis-backed SessionCache for tracking per-section token counts. The compose_context method is the single entry point that orchestrates the full assembly pipeline by calling private helpers _build_system_section, _build_memory_section, _build_history_section, and _apply_cache_markers.

1from dataclasses import dataclass, field
2from typing import Any
3import tiktoken
4
5@dataclass
6class BudgetConfig:
7    model_context_window: int = 200_000
8    output_reserve: int = 4_096
9    system_budget: int = 1_000
10    tools_budget: int = 5_000
11    memories_budget: int = 2_000
12    rag_budget: int = 8_000
13
14    @property
15    def history_budget(self) -> int:
16        fixed = (self.system_budget + self.tools_budget
17                 + self.memories_budget + self.rag_budget)
18        return self.model_context_window - self.output_reserve - fixed
19
20@dataclass
21class ComposedContext:
22    system: str
23    messages: list[dict[str, Any]]
24    tools: list[dict[str, Any]]
25    token_counts: dict[str, int] = field(default_factory=dict)
26
27class ContextComposer:
28    def __init__(self, budget: BudgetConfig, memory_client,
29                 session_factory, session_cache):
30        self.budget = budget
31        self.memory_client = memory_client
32        self.session_factory = session_factory
33        self.session_cache = session_cache
34        self._enc = tiktoken.get_encoding("cl100k_base")
35
36    def _count_tokens(self, text: str) -> int:
37        return len(self._enc.encode(text))
38
39    def _truncate_to_budget(self, text: str, budget: int) -> str:
40        tokens = self._enc.encode(text)
41        if len(tokens) <= budget:
42            return text
43        return self._enc.decode(tokens[:budget])

Assembling the Full Context and Attaching Cache Markers

The next code block implements the compose_context method together with _apply_cache_markers and build_api_payload. compose_context is the core orchestration function: it calls into Mem0 for memory retrieval, loads paginated conversation history from PostgreSQL via SQLAlchemy async sessions, enforces per-section budgets, and returns the fully assembled ComposedContext ready for the Anthropic API. The method also integrates with the Redis-backed SessionCache to persist per-section token counts so that subsequent requests can skip recomputation of stable sections. The companion _apply_cache_markers and build_api_payload methods then convert that ComposedContext into the final API payload, attaching {"cache_control": {"type": "ephemeral"}} to the stable prefix so Anthropic returns a cache hit on subsequent requests and charges only 10% of the normal input token cost for the cached prefix.

1async def compose_context(self, user_id: str, conversation_id: str,
2                          user_message: str, tools: list[dict],
3                          rag_results: list[str] | None = None
4                          ) -> ComposedContext:
5    system_text = self._build_system_section()
6    tool_schemas = self._truncate_tools(tools)
7
8    memories = self.memory_client.search(
9        query=user_message, user_id=user_id, limit=20
10    )
11    memory_block = self._build_memory_section(memories)
12
13    history = await self._load_history(conversation_id)
14    history = self._evict_history(history)
15
16    rag_block = ""
17    if rag_results is not None:
18        rag_block = "\n\n".join(rag_results)
19        rag_block = self._truncate_to_budget(
20            rag_block, self.budget.rag_budget
21        )
22
23    system_full = self._assemble_system(
24        system_text, memory_block, rag_block
25    )
26
27    messages = [*history, {"role": "user", "content": user_message}]
28
29    counts = {
30        "system": self._count_tokens(system_full),
31        "tools": self._count_tokens(str(tool_schemas)),
32        "memories": self._count_tokens(memory_block),
33        "history": sum(self._count_tokens(m["content"]) for m in messages),
34    }
35    await self.session_cache.store_counts(conversation_id, counts)
36
37    return ComposedContext(
38        system=system_full, messages=messages,
39        tools=tool_schemas, token_counts=counts,
40    )
41
42def _apply_cache_markers(self, system_text: str,
43                         tools: list[dict]) -> tuple[list[dict], list[dict]]:
44    system_blocks = [
45        {
46            "type": "text",
47            "text": system_text,
48            "cache_control": {"type": "ephemeral"},
49        }
50    ]
51
52    if tools:
53        tools[-1]["cache_control"] = {"type": "ephemeral"}
54
55    return system_blocks, tools
56
57def build_api_payload(self, composed: ComposedContext) -> dict:
58    system_blocks, tools = self._apply_cache_markers(
59        composed.system, composed.tools
60    )
61    return {
62        "model": "claude-sonnet-4-20250514",
63        "max_tokens": self.budget.output_reserve,
64        "system": system_blocks,
65        "tools": tools,
66        "messages": composed.messages,
67    }

Do's and Don'ts

Do's

1. ✓Do derive history_budget as a computed property of BudgetConfig — Calculate it as model_context_window - output_reserve - (system_budget + tools_budget + memories_budget + rag_budget) so conversation history automatically absorbs whatever the window has left; a hard-coded ceiling breaks silently whenever any other section's ceiling changes.
2. ✓Do attach cache_control: {"type": "ephemeral"} exclusively to the stable prefix (system prompt + tool schemas) — These two sections are identical across every request in a session, so marking the last block of that prefix lets Anthropic cache KV computations and cut input cost by up to 90% on cache hits; sections that vary per request — Mem0 memories, RAG chunks, history — must never carry the marker.
3. ✓Do target oldest conversation history turns first in the compose_context eviction loop — When the assembled total exceeds the model context window, discarding the tail of _build_history_section output preserves the system prompt, tool schemas, and Mem0 memories that encode long-term user preferences; those sections are far more expensive to reconstruct than older dialogue turns.

Don'ts

1. ✗Don't skip output_reserve when computing available context — BudgetConfig reserves 4,096 tokens for the model's generated output before dividing the remaining window among input sections; omitting that reserve causes history_budget to expand into the generation headroom, leaving the model no tokens to emit a response and producing silent truncation or API errors.
2. ✗Don't rely on rough character counts instead of _count_tokens with cl100k_base encoding — Claude's tokenizer splits on subwords, punctuation, and Unicode in ways that make character-based estimates diverge by 30–50% from real token counts; without calling tiktoken on each section before comparing it to its ceiling (memories_budget, tools_budget, etc.), sections silently overflow their budgets and trigger unnecessary history eviction.
3. ✗Don't apply cache_control to Mem0 memory results or RAG chunks — Because these sections are rebuilt from a semantic search on every user turn, their content changes with each request, making every cache lookup a miss; attaching {"cache_control": {"type": "ephemeral"}} to them wastes the cache slot without saving any compute, and can displace the stable-prefix cache entry that would have yielded the 90% cost reduction.