Build prompt versioning and A/B testing

Introduction

In production, prompt templates change frequently — tweaks for tone, adjustments for new model behavior, or experiments aimed at improving output quality. Without version control, it's impossible to know exactly what changed between deployments, roll back a regression, or run a controlled A/B test with confidence. Engineers who treat prompts as mutable strings rather than versioned artifacts accumulate invisible technical debt that surfaces as unexplained quality drops.

By the end of this lesson, you'll be able to build a PromptRegistry that tracks every version of a template, activates specific versions on demand, and maintains a change log suitable for audit and rollback.

Key Terminology

• PromptVersion — a Pydantic model that pairs a prompt template string with traceability metadata (version label, created_by, description, and a content hash), forming the atomic, immutable unit stored inside the registry.
• PromptRegistry — the container class that maintains the full version history for every named prompt in self.prompts, tracks which version is currently active in self.active, and appends every activation event to self._change_log.
• Content hash — an 8-character SHA-256 digest of the template text, computed automatically in model_post_init so that any change to a prompt's wording produces a visibly different fingerprint without requiring manual diffing.
• Change log — the append-only list _change_log that records each activation event with old_version, new_version, timestamp, and author, supplying the audit trail a rollback operation needs to recover a previous state.
• Active version — the version label stored in registry.active[name] that identifies which PromptVersion is currently serving; advanced by register whenever activate=True is passed.

Concepts

Prompts Are Code, Not Mutable Config

A prompt template looks like a string, but in production it behaves like a deployable artifact: it ships to a live model, its wording directly determines output quality, and a silent edit can degrade results across every downstream user in ways that are hard to trace. Treating a prompt as a mutable string — overwriting it in place when something needs to change — eliminates the ability to correlate a quality regression with its cause, roll back to a known-good state, or run a controlled experiment comparing two variants. The discipline shift is the same one that moved configuration into version control: the value is not just storing text, it's making every change attributable and reversible.

The Append-Only Registry with an Active Pointer

The key structural insight is to separate two distinct concerns: the complete history of all versions and which version is currently serving. PromptRegistry stores these separately — self.prompts[name] is a list that only ever grows (each register call appends a new PromptVersion; nothing is deleted or overwritten), while self.active[name] is a single version label that can be updated independently. Rolling back is therefore a cheap pointer operation: point active at an earlier entry in the list. The register method's activate parameter makes this explicit — you can add a new version to history without immediately promoting it, which is useful for staged rollouts or pre-loading candidates before a planned switch (see Code Walkthrough).

Content Hashing and the Change Log as an Audit Layer

Two mechanisms together answer the question "what changed, when, and who changed it." The content hash — an 8-character SHA-256 prefix computed automatically in model_post_init — acts as a fingerprint: if two PromptVersion objects share a hash, their templates are byte-for-byte identical; a single character difference produces a different hash. This catches accidental re-registrations and makes template drift visible at a glance without diffing raw strings.

The change log complements the hash by recording transitions rather than snapshots. Each entry captures old_version, new_version, timestamp, and the by field — exactly the context a rollback needs. When a quality regression surfaces, you can walk the log backward to find the last activation before the drop, read the hash of that version to confirm the template content, and re-activate it. Together, hash identity and a transition log give the registry the same auditability guarantees that a git history gives source code.

Code Walkthrough

Now that you understand how versioning discipline keeps prompt changes auditable and reversible, the registry below puts those ideas into working code. The PromptVersion model captures template content together with the metadata needed for traceability — who created it, when, and a short SHA-256 hash that makes content changes visible at a glance. The PromptRegistry wraps a collection of these versions, tracks which one is currently active, and logs every activation event.

Code snippetpython
1import hashlib
2from datetime import datetime
3from typing import Dict, List, Optional
4from pydantic import BaseModel, Field
5
6class PromptVersion(BaseModel):
7    version: str
8    template: str
9    description: str
10    created_at: datetime = Field(default_factory=datetime.now)
11    created_by: str
12    hash: str = ""
13    metadata: Dict[str, str] = Field(default_factory=dict)
14
15    def model_post_init(self, __context):
16        if not self.hash:
17            self.hash = hashlib.sha256(
18                self.template.encode()
19            ).hexdigest()[:8]
20
21class PromptRegistry:
22    def __init__(self):
23        self.prompts: Dict[str, List[PromptVersion]] = {}
24        self.active: Dict[str, str] = {}
25        self._change_log: List[Dict] = []
26
27    def register(
28        self,
29        name: str,
30        template: str,
31        description: str,
32        created_by: str,
33        activate: bool = True,
34        metadata: Optional[Dict[str, str]] = None,
35    ) -> PromptVersion:
36        if name not in self.prompts:
37            self.prompts[name] = []
38
39        version = f"v{len(self.prompts[name]) + 1}"
40        pv = PromptVersion(
41            version=version,
42            template=template,
43            description=description,
44            created_by=created_by,
45            metadata=metadata or {},
46        )
47        self.prompts[name].append(pv)
48
49        if activate:
50            old = self.active.get(name)
51            self.active[name] = version
52            self._change_log.append({
53                "timestamp": datetime.now().isoformat(),
54                "action": "activate",
55                "prompt": name,
56                "old_version": old,
57                "new_version": version,
58                "by": created_by,
59            })
60        return pv

model_post_init runs automatically after Pydantic validates the fields, so the content hash is always present without callers needing to compute it manually. register auto-increments the version label — v1, v2, and so on — so engineers never manage version strings by hand. The change log records the previous active version alongside the new one, which is exactly what a rollback operation needs later.

The following snippet exercises the registry end-to-end with two versions of the same prompt:

Code snippetpython
1registry = PromptRegistry()
2
3registry.register(
4    name="summarizer",
5    template="Summarize the following in three sentences: {{ text }}",
6    description="Initial version",
7    created_by="alice",
8)
9
10registry.register(
11    name="summarizer",
12    template="Summarize the following in two concise sentences: {{ text }}",
13    description="Shortened output for mobile",
14    created_by="bob",
15)
16
17print(registry.active["summarizer"])            # v2
18print(len(registry.prompts["summarizer"]))      # 2
19print(registry._change_log[-1]["old_version"])  # v1

The second register call does not overwrite the first — both versions remain in registry.prompts["summarizer"], and the change log captures the transition so the previous state is always recoverable. Confirm that registry.active["summarizer"] prints "v2" and that the final change-log entry shows old_version as "v1" — those two assertions together verify that the registry is tracking version transitions correctly.

Do's and Don'ts

Having walked through building prompt versioning above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do pass activate=False when registering a draft version — storing a new PromptVersion in the registry without immediately flipping self.active lets you stage experimental templates alongside the live one, so A/B tests and canary rollouts don't disrupt production traffic the moment the template is written.
2. ✓Do rely on model_post_init to compute the SHA-256 hash automatically — because PromptVersion derives hash from self.template.encode() during Pydantic initialization, the eight-character fingerprint is always consistent with actual template content; manually setting or copying hash values bypasses this guarantee and makes content-change detection unreliable.
3. ✓Do inspect _change_log[-1]["old_version"] before performing a rollback — the change log records both old_version and new_version on every activate event, so you always have the exact version string needed to call registry.active[name] = old_version and restore the previous state without guessing.

Don'ts

1. ✗Don't mutate a PromptVersion.template field after registration — the SHA-256 hash is computed once in model_post_init and stored; editing the template string afterward causes the stored hash to no longer match the content, silently breaking any diff or content-change audit that relies on it.
2. ✗Don't manage version strings like "v2" by hand — register derives the label from f"v{len(self.prompts[name]) + 1}" automatically; hard-coding or skipping version labels breaks the sequential numbering that _change_log entries and rollback logic depend on.
3. ✗Don't treat the registry's active dict as the complete prompt history — self.active holds only the currently active version string per name; the full list of every registered template lives in self.prompts[name], and discarding or replacing that list destroys the rollback chain the change log points into.