Build eval suite versioning and management

Introduction

Engineers often discover model regressions weeks after a deployment, only to find that the eval suite configuration has since changed — making it impossible to reconstruct the exact test cases, assertion rubrics, and model configs that originally cleared the release gate. Without a system that treats eval suites as immutable, versioned artifacts with a tracked promotion lifecycle, diagnosing what went wrong collapses into guesswork. By the end of this lesson, you'll be able to build an eval pipeline management system that versions suites through structured lifecycle states, enforces that only ACTIVE suites can gate CI/CD deployments, and produces a diff view between any two versions so reviewers can see precisely what changed before approving a promotion.

Key Terminology

• EvalSuiteVersion — A dataclass that captures the complete, point-in-time state of an evaluation suite, bundling test_cases, assertion_configs, and model_configs together with a lifecycle status and an immutable version_id derived from the suite's content hash.
• SuiteStatus — An enum with four values (DRAFT, REVIEWED, ACTIVE, ARCHIVED) that tracks where an eval suite sits in its promotion lifecycle; only suites in the ACTIVE state can gate CI/CD production deployments.
• content hash — A deterministic version identifier produced by EvalSuiteVersion.content_hash(), computed as the SHA-256 of the suite's test cases, assertions, and model configs serialized with sorted keys; identical content always yields the same hash regardless of author or creation time.
• VALID_TRANSITIONS — A dictionary that maps each SuiteStatus to its permitted successor states, enforcing the promotion workflow and preventing lifecycle violations such as reactivating an ARCHIVED suite or skipping the review step.
• EvalSuiteManager — The orchestration class responsible for creating new DRAFT versions, promoting versions between lifecycle states, and generating diff reports between two EvalSuiteVersion objects, persisting each version as a JSON file under a configurable storage_dir.
• diff view — A structured comparison between two EvalSuiteVersion objects that enumerates which test cases were added, removed, or modified, giving reviewers a precise machine-generated change report during suite promotion.

Concepts

Eval Suites as First-Class Versioned Artifacts

Checking an evaluation suite into Git as a YAML config file is necessary but not sufficient. When a regression is detected six weeks after a model ships, you need to reconstruct the exact evaluation environment that cleared it: which test cases, which assertion rubrics, and which model configurations were bundled together at approval time. Git records file diffs, but it does not record which version of an eval suite was ACTIVE in CI/CD at the moment a specific model was promoted. The EvalSuiteVersion dataclass closes this gap by treating each snapshot of the suite as an immutable artifact — a cryptographic commitment to its contents — rather than a mutable file that any commit can silently alter.

This shifts the mental model from "the eval suite is a config" to "the eval suite is a releasable artifact with its own lifecycle." Just as a compiled binary has a build hash and a deploy status, an EvalSuiteVersion has a content_hash-derived version_id and a SuiteStatus. Both are facts that need to be tracked and audited independently of the pipeline that runs them.

The Promotion Lifecycle and Why Only ACTIVE Suites Can Gate Deployments

The four SuiteStatus states — DRAFT, REVIEWED, ACTIVE, ARCHIVED — mirror the code-promotion pipeline that software teams already use for releases. A new suite starts as DRAFT while under development, advances to REVIEWED after peer approval, becomes ACTIVE when it is the live gate in CI/CD, and finally moves to ARCHIVED when a newer version supersedes it. The critical invariant is that only ACTIVE suites can gate production deployments; a DRAFT that has never been reviewed, or an ARCHIVED suite that has been superseded, cannot silently approve a model that should have been caught.

The VALID_TRANSITIONS dictionary enforces this by making certain moves illegal at the data level (see Code Walkthrough). An ARCHIVED suite has no valid successors, so reactivation is impossible. A DRAFT can only move to REVIEWED, never directly to ACTIVE, ensuring the peer-review step cannot be bypassed under time pressure. These constraints mean the lifecycle violations that tend to slip through in informal processes — "we'll review it after it ships" — are structurally prevented.

Content-Addressed Versioning and Diff-Driven Review

EvalSuiteVersion.content_hash() derives the version ID deterministically from the suite's test cases, assertions, and model configs, serialized with sorted keys before hashing. Two contributors independently assembling the same suite content produce the same version_id; any change to even one test case produces a different hash. This eliminates ambiguous human-assigned labels like "v1.1b" in favor of identifiers that are computed from, and therefore coupled to, the actual content.

When a suite moves from DRAFT to REVIEWED, the diff view gives reviewers a structured change report — which test cases were added, which were removed, and which were modified — rather than asking them to manually compare raw JSON files. This is especially important in large suites where a targeted addition of edge-case prompts for a newly discovered failure mode should be clearly distinguishable from a wholesale rewrite. The diff report is the paper trail that connects the promotion decision to the specific changes being approved.

Code Walkthrough

Now that you understand why eval suites must be treated as releasable artifacts with their own immutable snapshots and promotion lifecycle, the classes below put that model into working code.

SuiteStatus encodes the four states a suite can occupy — DRAFT while under development, REVIEWED after peer approval, ACTIVE when gating CI/CD, and ARCHIVED when retired. VALID_TRANSITIONS enforces the one-way promotion path: a DRAFT may only advance to REVIEWED; a REVIEWED suite may move to ACTIVE or revert to DRAFT; an ACTIVE suite can only be ARCHIVED; and ARCHIVED suites have no permitted successors.

EvalSuiteVersion captures the complete point-in-time snapshot — test cases, assertion configs, and model configs — and derives a deterministic version_id from a SHA-256 content hash so identical content always produces the same identifier regardless of author or creation time. EvalSuiteManager owns the create-promote-diff workflow: create_version initializes a new DRAFT and persists it as a JSON file; promote validates the requested transition against VALID_TRANSITIONS before updating status and recording who promoted it and when; diff_versions compares two EvalSuiteVersion objects and returns which test cases were added, removed, or modified.

Code snippetpython
1import json
2import hashlib
3from datetime import datetime, timezone
4from pathlib import Path
5from dataclasses import dataclass, field, asdict
6from enum import Enum
7
8class SuiteStatus(Enum):
9    DRAFT = "draft"
10    REVIEWED = "reviewed"
11    ACTIVE = "active"
12    ARCHIVED = "archived"
13
14VALID_TRANSITIONS = {
15    SuiteStatus.DRAFT: [SuiteStatus.REVIEWED],
16    SuiteStatus.REVIEWED: [SuiteStatus.ACTIVE, SuiteStatus.DRAFT],
17    SuiteStatus.ACTIVE: [SuiteStatus.ARCHIVED],
18    SuiteStatus.ARCHIVED: [],
19}
20
21@dataclass
22class EvalSuiteVersion:
23    version_id: str
24    suite_name: str
25    status: SuiteStatus
26    created_at: str
27    test_cases: list[dict]
28    assertion_configs: list[dict]
29    model_configs: list[dict]
30    created_by: str = ""
31    promoted_by: str = ""
32    promoted_at: str = ""
33
34    def content_hash(self) -> str:
35        content = json.dumps(
36            {"tests": self.test_cases, "assertions": self.assertion_configs, "models": self.model_configs},
37            sort_keys=True,
38        )
39        return hashlib.sha256(content.encode()).hexdigest()[:12]
40
41@dataclass
42class EvalSuiteManager:
43    storage_dir: Path = field(default_factory=lambda: Path("eval_suites"))
44
45    def __post_init__(self):
46        self.storage_dir.mkdir(parents=True, exist_ok=True)
47
48    def create_version(self, suite_name, test_cases, assertion_configs, model_configs, created_by=""):
49        now = datetime.now(timezone.utc).isoformat()
50        version = EvalSuiteVersion(
51            version_id="", suite_name=suite_name, status=SuiteStatus.DRAFT,
52            created_at=now, test_cases=test_cases, assertion_configs=assertion_configs,
53            model_configs=model_configs, created_by=created_by,
54        )
55        version.version_id = version.content_hash()
56        path = self.storage_dir / f"{suite_name}_{version.version_id}.json"
57        path.write_text(json.dumps(asdict(version), indent=2, default=str))
58        return version
59
60    def promote(self, version, target_status, promoted_by=""):
61        allowed = VALID_TRANSITIONS[version.status]
62        if target_status not in allowed:
63            raise ValueError(f"Cannot transition {version.status} → {target_status}. Allowed: {allowed}")
64        version.status = target_status
65        version.promoted_by = promoted_by
66        version.promoted_at = datetime.now(timezone.utc).isoformat()
67        path = self.storage_dir / f"{version.suite_name}_{version.version_id}.json"
68        path.write_text(json.dumps(asdict(version), indent=2, default=str))
69        return version
70
71    def diff_versions(self, old: "EvalSuiteVersion", new: "EvalSuiteVersion") -> dict:
72        old_ids = {tc["id"]: tc for tc in old.test_cases}
73        new_ids = {tc["id"]: tc for tc in new.test_cases}
74        return {
75            "added":    [tc for id_, tc in new_ids.items() if id_ not in old_ids],
76            "removed":  [tc for id_, tc in old_ids.items() if id_ not in new_ids],
77            "modified": [{"old": old_ids[id_], "new": tc}
78                         for id_, tc in new_ids.items() if id_ in old_ids and old_ids[id_] != tc],
79        }

The second block walks the full promotion path and shows how to use the diff view before approving a version for CI/CD gating.

Code snippetpython
1manager = EvalSuiteManager(storage_dir=Path("eval_suites"))
2
3v1 = manager.create_version(
4    suite_name="summarization-eval",
5    test_cases=[{"id": "tc-001", "prompt": "Summarize this article.", "expected": "concise summary"}],
6    assertion_configs=[{"type": "similarity", "threshold": 0.85}],
7    model_configs=[{"provider": "openai", "model": "gpt-4o"}],
8    created_by="ml-engineer",
9)
10print(f"Created {v1.version_id} [{v1.status.value}]")
11manager.promote(v1, SuiteStatus.REVIEWED, promoted_by="tech-lead")
12
13# Add a test case and diff before promoting the new version
14v2 = manager.create_version(
15    suite_name="summarization-eval",
16    test_cases=[
17        {"id": "tc-001", "prompt": "Summarize this article.", "expected": "concise summary"},
18        {"id": "tc-002", "prompt": "Summarize a news report.", "expected": "brief overview"},
19    ],
20    assertion_configs=[{"type": "similarity", "threshold": 0.88}],
21    model_configs=[{"provider": "openai", "model": "gpt-4o"}],
22    created_by="ml-engineer",
23)
24
25diff = manager.diff_versions(v1, v2)
26print(f"Added: {len(diff['added'])}  Removed: {len(diff['removed'])}  Modified: {len(diff['modified'])}")
27
28manager.promote(v2, SuiteStatus.REVIEWED, promoted_by="tech-lead")
29manager.promote(v2, SuiteStatus.ACTIVE, promoted_by="release-bot")
30print(f"Active suite: {v2.suite_name} @ {v2.version_id}")

Confirm that JSON files appear under eval_suites/, that attempting to promote a DRAFT directly to ACTIVE raises a ValueError, and that the diff report lists exactly one added entry for tc-002 with zero removed or modified cases.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do derive version_id from a SHA-256 content hash of test_cases, assertion_configs, and model_configs — this makes identical suite content produce the same identifier regardless of who created it or when, so you can detect duplicate submissions and reconstruct any historical snapshot deterministically.
2. ✓Do enforce VALID_TRANSITIONS in promote() before mutating status — the one-way path (DRAFT → REVIEWED → ACTIVE → ARCHIVED) ensures that only a REVIEWED suite can reach ACTIVE and gate CI/CD, making it impossible to bypass peer approval by promoting directly from DRAFT.
3. ✓Do call diff_versions() between two EvalSuiteVersion objects before approving a promotion to ACTIVE — the added/removed/modified breakdown on test_cases gives reviewers an exact record of what changed, preventing silent rubric drift from reaching the release gate.

Don'ts

1. ✗Don't skip persisting the full EvalSuiteVersion snapshot (including assertion_configs and model_configs) to a JSON file at creation time — storing only test cases omits the assertion thresholds and model configs that define the gate, making it impossible to reproduce the exact conditions that cleared a past release.
2. ✗Don't allow a suite in any status other than ACTIVE to gate CI/CD deployments — permitting REVIEWED or DRAFT suites to block or pass builds defeats the promotion lifecycle and lets unapproved rubric changes reach production.
3. ✗Don't compute version_id before populating test_cases, assertion_configs, and model_configs on the EvalSuiteVersion object — hashing an incomplete or default-valued payload produces a collision-prone identifier that no longer uniquely fingerprints the suite's real content, as create_version demonstrates by setting version_id = "" first and calling content_hash() only after all fields are assigned.