Design A/B experiments for prompt variants

Introduction

When you change a prompt template, swap a model checkpoint, or alter a RAG retrieval strategy, "looks better on a handful of examples" is not a defensible decision—LLM outputs are stochastic, high-dimensional, and judged on subjective quality metrics that can move in opposite directions for the same edit. Teams that skip rigorous experiment design routinely ship "improvements" that quietly raise hallucination rates or degrade fluency, because they peeked at a 100-request sample and saw what they wanted to see.

A/B testing in traditional web applications measures button clicks and page views—discrete, deterministic events. LLM systems demand more: structured experiment definitions, deterministic user-to-variant assignment, and statistically grounded sample size planning. Get any of these wrong and you will either fail to detect real regressions that ship harm to production, or chase noise as if it were signal and waste engineering cycles on no-op changes.

By the end of this lesson you'll be able to define an experiment with typed control and treatment configurations, assign users deterministically with consistent hashing so one user always sees one variant, and calculate the minimum sample size required to detect a meaningful effect with adequate statistical power.

Key Terminology

• Control group: The baseline variant that serves the current production configuration (prompt, model, or RAG pipeline) against which all treatment variants are compared.
• Treatment group: The experimental variant containing exactly one changed variable—a modified prompt template, a different model checkpoint, or an altered retrieval strategy.
• Randomization unit: The entity (user, session, or request) that gets deterministically assigned to a variant; choosing the wrong unit inflates false-positive rates through correlated observations.
• Effect size: The magnitude of the difference between control and treatment on a target metric, expressed as Cohen's d for continuous metrics or as a proportion difference for binary outcomes.
• Guardrail metric: A safety-critical metric (toxicity rate, hallucination frequency, latency p99) that must not degrade beyond a predefined threshold regardless of the primary metric outcome.
• Statistical power: The probability of correctly detecting a true effect when one exists; conventionally set at 0.80, meaning a 20% chance of a false negative.

Concepts

A/B experimentation turns "did this prompt change help?" into a measurable comparison by serving two variants side by side and deciding the winner with statistics rather than intuition.

Control, treatment, and sticky assignment

An experiment pins down two configurations: the control (today's prompt, model, or settings) and the treatment (the proposed change). Holding everything else constant, any difference in outcomes between the two groups is attributable to the change itself — that is the whole point of a control. To keep the comparison clean, each user must be assigned to exactly one group and stay there for the life of the experiment.

Deterministic, hash-based assignment makes that stickiness free of any stored state: hashing the user and experiment identifiers together and mapping the result into a fixed bucket (for example, mod 100 compared against the traffic percentage) yields the same variant for the same user every time, while distributing users evenly and independently across experiments. A user never flickers between control and treatment, so their experience is consistent and their measured behavior belongs cleanly to one group.

Guardrails and pre-sized samples

Two safeguards keep an experiment both safe and honest. Guardrail metrics — latency, error rate, cost, refusal rate — are monitored continuously, and a breach trips an automatic stop so a bad treatment is pulled before it harms many users; the lifecycle moves from running to stopped without waiting for a human.

The minimum sample size is calculated in advance from the effect you want to detect and your tolerance for error, then frozen. Fixing it upfront prevents peeking: repeatedly checking results and calling a winner the moment they look significant dramatically inflates false-positive rates, because random noise will eventually cross the threshold by chance. By committing to evaluate only once the planned sample is reached, the experiment preserves the statistical guarantees its conclusion depends on.

Code Walkthrough

Structuring the Experiment Definition

Every LLM experiment must capture five elements: what you are testing (the hypothesis), what stays constant (the control configuration), what changes (the treatment configuration), how you measure success (primary and guardrail metrics), and how much traffic you allocate to each group. Encoding these in a typed data model prevents the configuration drift and ambiguity that plague ad-hoc experimentation.

The following code defines an ExperimentConfig dataclass and an Experiment Pydantic model that together form the backbone of a structured experimentation framework. The ExperimentConfig class holds the prompt template and model identifier for a single variant, while the Experiment model captures the full experimental design including the hypothesis string, traffic allocation ratio, a list of metric names to track, and a list of guardrail metric names that serve as safety constraints. The ExperimentStatus enum tracks lifecycle state transitions from "draft" through "running" to "completed" or "stopped".

1from pydantic import BaseModel, Field, field_validator
2from enum import Enum
3from datetime import datetime
4from typing import Optional
5import uuid
6
7class ExperimentStatus(str, Enum):
8    DRAFT = "draft"
9    RUNNING = "running"
10    COMPLETED = "completed"
11    STOPPED = "stopped"
12
13class ExperimentConfig(BaseModel):
14    prompt_template: str
15    model_id: str = "gpt-4o"
16    temperature: float = Field(default=0.7, ge=0.0, le=2.0)
17    max_tokens: int = Field(default=1024, gt=0)
18    rag_top_k: Optional[int] = Field(default=None, ge=1)
19
20class Experiment(BaseModel):
21    experiment_id: str = Field(default_factory=lambda: str(uuid.uuid4()))
22    name: str = Field(..., min_length=3, max_length=120)
23    hypothesis: str = Field(..., min_length=10)
24    control_config: ExperimentConfig
25    treatment_config: ExperimentConfig
26    traffic_percentage: float = Field(default=50.0, gt=0, le=50)
27    primary_metrics: list[str] = Field(default=["quality_score"])
28    guardrail_metrics: list[str] = Field(
29        default=["toxicity_rate", "hallucination_rate", "latency_p99"]
30    )
31    status: ExperimentStatus = ExperimentStatus.DRAFT
32    created_at: datetime = Field(default_factory=datetime.utcnow)
33    min_sample_size: Optional[int] = None
34
35    @field_validator("treatment_config")
36    @classmethod
37    def configs_must_differ(cls, v, info):
38        control = info.data.get("control_config")
39        if control and v == control:
40            raise ValueError("Treatment must differ from control")
41        return v

Experiment Lifecycle Flow

The following diagram illustrates how an experiment transitions through its lifecycle states and how individual requests flow through the assignment and evaluation pipeline.

This lifecycle enforces three critical safeguards. First, the experiment cannot start until configuration validation passes and a minimum sample size has been calculated. Second, every logged observation triggers a guardrail check—if any safety metric breaches its threshold, the experiment is automatically stopped before further harm occurs. Third, the experiment only transitions to "completed" after accumulating the pre-calculated minimum sample size, preventing premature peeking that inflates false-positive rates.

Deterministic Assignment and Sample Size Planning

Naive random assignment using Python's random.choice introduces two problems: a user might see the control prompt on their first request and the treatment prompt on their second, creating a confusing experience and violating the independence assumption required for valid statistical inference. Consistent hashing solves both problems by mapping the combination of user_id and experiment_id to a deterministic bucket. Equally critical is committing to a pre-calculated sample size: engineers who launch an experiment, check results daily, and stop as soon as the p-value drops below 0.05—a practice called "peeking"—can inflate the actual false-positive rate to 20–30%. The combined snippet below addresses both: assign_variant uses SHA-256 to map (user_id, experiment_id) to a stable bucket, and calculate_min_sample_size uses the two-proportion z-test formula to determine how many observations each group needs before any results are read.

Code snippetpython
1import hashlib
2import math
3from scipy.stats import norm
4
5def assign_variant(user_id: str, experiment: Experiment) -> str:
6    """Deterministically assign a user to control or treatment."""
7    hash_input = f"{user_id}:{experiment.experiment_id}".encode("utf-8")
8    hash_bytes = hashlib.sha256(hash_input).digest()
9    bucket = int.from_bytes(hash_bytes[:8], byteorder="big") % 100
10    if bucket < experiment.traffic_percentage:
11        return "treatment"
12    return "control"
13
14def get_config_for_user(
15    user_id: str, experiment: Experiment
16) -> tuple[str, ExperimentConfig]:
17    """Return (variant_name, config) for a given user."""
18    variant = assign_variant(user_id, experiment)
19    if variant == "treatment":
20        return variant, experiment.treatment_config
21    return variant, experiment.control_config
22
23def verify_split_balance(
24    experiment: Experiment, user_ids: list[str]
25) -> dict[str, float]:
26    """Verify actual traffic split matches expected percentage."""
27    assignments = [assign_variant(uid, experiment) for uid in user_ids]
28    treatment_count = assignments.count("treatment")
29    actual_pct = (treatment_count / len(user_ids)) * 100
30    return {
31        "expected_pct": experiment.traffic_percentage,
32        "actual_pct": round(actual_pct, 2),
33        "deviation": round(abs(actual_pct - experiment.traffic_percentage), 2),
34        "total_users": len(user_ids),
35    }
36
37def calculate_min_sample_size(
38    baseline_rate: float,
39    min_detectable_effect: float,
40    alpha: float = 0.05,
41    power: float = 0.80,
42) -> dict[str, int | float]:
43    """Calculate minimum sample size per group for a two-proportion test."""
44    p1 = baseline_rate
45    p2 = baseline_rate + min_detectable_effect
46    p_avg = (p1 + p2) / 2.0
47
48    z_alpha = norm.ppf(1 - alpha / 2)
49    z_beta = norm.ppf(power)
50
51    numerator = (
52        z_alpha * math.sqrt(2 * p_avg * (1 - p_avg))
53        + z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
54    ) ** 2
55    denominator = (p2 - p1) ** 2
56
57    n_per_group = math.ceil(numerator / denominator)
58
59    cohens_h = 2 * math.asin(math.sqrt(p2)) - 2 * math.asin(math.sqrt(p1))
60
61    return {
62        "n_per_group": n_per_group,
63        "n_total": n_per_group * 2,
64        "baseline_rate": p1,
65        "target_rate": p2,
66        "effect_size_h": round(abs(cohens_h), 4),
67        "alpha": alpha,
68        "power": power,
69    }

• Imports: hashlib supplies SHA-256 (uniform distribution across buckets with negligible collision risk), math supplies ceiling and square-root, and scipy.stats.norm supplies the inverse CDF (ppf) of the standard normal distribution.
• assign_variant: Encodes the concatenation of user_id and experiment_id into bytes, computes the SHA-256 digest, extracts the first 8 bytes as a big-endian integer, and takes modulo 100 to produce a bucket in the range 0–99. If the bucket is less than traffic_percentage, the user receives the treatment; otherwise, the control.
• get_config_for_user: Returns a tuple of the variant name string and the corresponding ExperimentConfig, providing a single call site for the serving layer to obtain both the assignment label (needed for logging) and the actual configuration (needed for execution).
• verify_split_balance: A diagnostic utility that takes a list of user IDs, computes assignments for all of them, and returns the expected versus actual traffic percentage along with the deviation. Run it against a sample of 10,000 historical user IDs to confirm the hash function produces a balanced split before any real traffic is exposed.
• calculate_min_sample_size signature: Requires the baseline_rate (e.g., 0.72 if 72% of current responses pass quality review) and the min_detectable_effect (e.g., 0.05 to detect a 5 percentage-point improvement). Both alpha and power default to standard values (0.05 and 0.80).
• z-values and formula: z_alpha is the critical value for a two-sided test at significance level alpha (≈1.96 at α=0.05); z_beta is the critical value for the desired power (≈0.84 at power=0.80). The numerator squares the sum of the null-hypothesis standard error scaled by z_alpha and the alternative-hypothesis standard error scaled by z_beta; the denominator is the squared difference between the two proportions.
• math.ceil: Rounds up because you cannot run a fraction of an observation; under-powering an experiment by even one sample technically violates the design contract.
• Cohen's h: Provides a standardized effect size for proportion comparisons using the arcsine transformation, making it comparable across experiments with different baseline rates. An h of 0.2 is conventionally considered small, 0.5 medium, and 0.8 large.

The critical property of the hashing approach is that the same user_id and experiment_id combination always produces the same bucket value. A user who makes 50 requests during an experiment always sees the same variant. Equally important, including the experiment_id in the hash input means that a user assigned to treatment in Experiment A has an independent, unbiased assignment in Experiment B—preventing carry-over effects when running concurrent experiments.

As a concrete example, if your current prompt produces quality-passing responses at a baseline_rate of 0.70 and you want to detect a 5 percentage-point improvement (min_detectable_effect=0.05), this function returns approximately 1,237 observations per group—2,474 total. At 500 daily requests split 50/50, that experiment requires roughly 10 days of traffic. This kind of upfront calculation prevents the two most common experiment failures: stopping too early (false positives from peeking) and running too long (wasting engineering cycles on an experiment that already has sufficient data).

Wiring min_sample_size back into the Experiment model closes the loop: set experiment.min_sample_size = result["n_per_group"] before transitioning to the "running" state, and the experiment runner refuses to compute final statistics until both groups have accumulated at least that many observations. This design pattern—pre-registration of the sample size before data collection begins—is the single most effective guardrail against the statistical pitfalls that undermine LLM experimentation at scale.

Do's and Don'ts

Do's

1. ✓Do encode both control_config and treatment_config as typed ExperimentConfig objects with a field_validator that rejects identical configs — running an experiment where treatment equals control wastes traffic allocation and produces statistically meaningless results with no hypothesis being tested.
2. ✓Do cap traffic_percentage at 50.0 in the Experiment model — keeping the majority of users on the proven control path limits blast radius if the treatment degrades guardrail metrics like hallucination_rate or latency_p99 before the minimum sample size is reached.
3. ✓Do assign users by hashing user_id + experiment_id and taking the result modulo 100 — deterministic consistent hashing guarantees one user always sees one variant across repeated requests, preventing the contamination of results that occurs when the same user experiences both control and treatment prompts.

Don'ts

1. ✗Don't evaluate a prompt or model change on a small convenience sample and treat directional movement as confirmation — LLM outputs are stochastic and quality metrics can move in opposite directions on the same edit; the min_sample_size field exists precisely to enforce statistically powered observation before drawing conclusions.
2. ✗Don't leave the hypothesis field as a vague string or skip it — the 10-character minimum is a floor, not a target; an undocumented hypothesis makes it impossible to distinguish a pre-registered prediction from post-hoc rationalization of whichever variant happened to win.
3. ✗Don't ignore guardrail metrics (toxicity_rate, hallucination_rate, latency_p99) in favor of optimizing only primary_metrics — a treatment that improves quality score while silently raising hallucination rate is a regression, and the lifecycle's guardrail-violation check is the only automated gate that stops such an experiment before it completes.