Build eval regression root cause analyzer

Introduction

When you run eval comparisons across model versions, a 1% aggregate drop in pass rate tells you almost nothing about what actually broke. The regression could be hiding in a single domain, a specific prompt class, or even a grader update that shifted scoring distributions — yet the top-line number masks all three simultaneously.

By the end of this lesson, you will have built a root-cause analyzer that segments eval results across domain, prompt class, and prompt length, ranks each segment by statistical significance and effect size, and surfaces the top drivers of any regression in a structured CI-ready report.

Key Terminology

• segment_key — A tuple of axis values (e.g., ("legal", "instruction", "long")) that uniquely identifies one cross-product cell in the segmentation; each element corresponds to one entry in segment_axes, so the tuple's length matches the number of configured axes.
• segment_axes — The ordered list of field names used to partition per-test results into cells; the default ["domain", "class", "length_bucket"] is the minimum useful resolution, and every input record in baseline_results or candidate_results must carry a field for each named axis.
• Chi-square contingency test — A statistical test applied to a 2 × 2 table of (passes, failures) from baseline and candidate runs to determine whether the observed pass-rate difference in a cell is unlikely to be random noise; implemented via scipy.stats.chi2_contingency in _chi_square().
• Cohen's h — An effect-size measure for the difference between two proportions, computed as 2 × (arcsin(√p₁) − arcsin(√p₂)) in _cohen_h(); values near 0.2 indicate a small effect, 0.5 medium, and 0.8 large, giving magnitude independent of sample size.
• rank_score — The composite priority score stored in each SegmentResult, computed as |delta| × (1 − p_value) × |effect_size|; it rewards segments that are simultaneously large in absolute pass-rate change, statistically significant, and meaningful in magnitude.
• Minimum cell size — The 5-sample floor applied in analyze() before any statistical test: cells with fewer than five results in either run are dropped because chi-square contingency tests produce unreliable p-values at very small counts.

Concepts

Why Aggregate Pass Rates Cannot Identify Root Causes

A top-line pass rate is a weighted average across every test case in the run. When a model update causes one domain to regress sharply while another improves, the effects partially cancel — and the aggregate drop is smaller than either movement in isolation. Three structurally distinct failure modes produce indistinguishable aggregate signals: a domain regression (legal prompts break while medical holds), a prompt-class regression (instruction-following degrades while open-ended generation is unchanged), and a grader-induced regression (a scoring rule change shifts the distribution with no model change at all). No top-line number distinguishes between them. Investigating without segmentation means forming a hypothesis blind, then testing it by hand — the slowest possible debugging loop.

Cross-Product Segmentation: Cells, Not Slices

Slicing along a single axis — "how did domain X perform?" — misses interaction effects. A test might belong to the medical domain, the instruction class, and the long length bucket simultaneously. If only the combination of all three regresses, neither a domain-only nor a class-only slice surfaces it. RegressionAnalyzer forms the cross-product of all axes listed in segment_axes, creating one cell per unique tuple and routing every test record into exactly one cell via _group().

The 5-sample minimum per cell is not an arbitrary threshold — chi-square contingency tests produce unreliable p-values when observed counts are very small, and a cell with three observations is not a sound basis for a regression claim. Any cell below that floor is dropped before statistical testing begins (see Code Walkthrough).

Ranking by a Composite of Delta, Significance, and Effect Size

No single statistic is sufficient on its own for prioritizing segments. A large dataset will render a 0.001 pass-rate difference "statistically significant" (small p-value) even though no engineer should act on it. Conversely, a large Cohen's h in a noisy, under-sampled segment may be a statistical artifact. Absolute delta ignores sample size and significance entirely.

rank_score = |delta| × (1 − p_value) × |effect_size| multiplies all three factors together so a segment earns a high score only when the pass-rate gap is large in absolute terms, the chi-square test says the difference is unlikely due to chance, and Cohen's h confirms the effect is meaningful in magnitude. render_report then caps output at five segments — beyond five, the report exceeds a reader's working-memory budget and the signal-to-noise ratio inverts. When every segment's rank_score falls below 0.001, the report short-circuits to a clean "no significant segment-level regression detected" message rather than emitting an empty or misleading table (see Code Walkthrough).

Code Walkthrough

Now that you understand why segment regression, prompt-class regression, and grader-induced regression all hide inside aggregate numbers — and why crossing at least three axes simultaneously is the minimum useful resolution — the following implementation encodes that structure directly.

The RegressionAnalyzer groups per-test results into cross-product cells, tests each cell for significance, and ranks by a composite of delta, p-value, and effect size:

Code snippetpython
1import math
2from collections import defaultdict
3from dataclasses import dataclass, field
4from scipy import stats
5
6@dataclass
7class SegmentResult:
8    segment_key: tuple
9    n_baseline: int
10    n_candidate: int
11    pass_baseline: float
12    pass_candidate: float
13    p_value: float
14    effect_size: float
15    rank_score: float = 0.0
16
17@dataclass
18class RegressionAnalyzer:
19    baseline_results: list[dict]
20    candidate_results: list[dict]
21    segment_axes: list[str] = field(
22        default_factory=lambda: ["domain", "class", "length_bucket"]
23    )
24
25    def analyze(self) -> list[SegmentResult]:
26        b = self._group(self.baseline_results)
27        c = self._group(self.candidate_results)
28        out = []
29        for key in sorted(set(b) | set(c)):
30            n_b, n_c = len(b.get(key, [])), len(c.get(key, []))
31            if n_b < 5 or n_c < 5:
32                continue
33            pass_b = sum(b[key]) / n_b
34            pass_c = sum(c[key]) / n_c
35            chi2, p = self._chi_square(b[key], c[key])
36            h = self._cohen_h(pass_b, pass_c)
37            rank = abs(pass_c - pass_b) * (1 - p) * abs(h)
38            out.append(SegmentResult(key, n_b, n_c, pass_b, pass_c, p, h, rank))
39        out.sort(key=lambda s: -s.rank_score)
40        return out
41
42    def _group(self, results: list[dict]) -> dict:
43        groups = defaultdict(list)
44        for r in results:
45            key = tuple(r.get(a, "?") for a in self.segment_axes)
46            groups[key].append(int(bool(r["pass"])))
47        return groups
48
49    @staticmethod
50    def _chi_square(a, b):
51        passes_a, fails_a = sum(a), len(a) - sum(a)
52        passes_b, fails_b = sum(b), len(b) - sum(b)
53        try:
54            chi2, p, _, _ = stats.chi2_contingency(
55                [[passes_a, fails_a], [passes_b, fails_b]]
56            )
57            return chi2, p
58        except ValueError:
59            return 0.0, 1.0
60
61    @staticmethod
62    def _cohen_h(p1, p2) -> float:
63        return 2 * (math.asin(math.sqrt(p1)) - math.asin(math.sqrt(p2)))

Each record in baseline_results and candidate_results must carry a "pass" boolean and a field for each axis named in segment_axes. The analyzer skips any cell with fewer than five samples in either run — the practical minimum for a reliable chi-square — then ranks surviving cells by |delta| × (1 − p) × |effect_size|, rewarding segments that are large in absolute change, statistically real, and meaningful in magnitude. Cohen's h values near 0.2 indicate a small effect; 0.5 medium; 0.8 large.

Once analyze() returns a ranked list, pass it to render_report for CI output:

Code snippetpython
1def render_report(results: list[SegmentResult], top: int = 5) -> str:
2    lines = ["# Regression Root-Cause Report", ""]
3    if not results or results[0].rank_score < 0.001:
4        lines += ["**No significant segment-level regression detected.**"]
5        return "\n".join(lines)
6    lines += [f"**Top {top} segments by impact:**", ""]
7    lines += ["| Segment | n_b | n_c | base | cand | Δ | p | Cohen's h |"]
8    lines += ["|---|---|---|---|---|---|---|---|"]
9    for r in results[:top]:
10        delta = r.pass_candidate - r.pass_baseline
11        sign = "+" if delta >= 0 else ""
12        lines.append(
13            f"| {r.segment_key} | {r.n_baseline} | {r.n_candidate} "
14            f"| {r.pass_baseline:.3f} | {r.pass_candidate:.3f} "
15            f"| {sign}{delta:.3f} | {r.p_value:.4f} | {r.effect_size:+.3f} |"
16        )
17    return "\n".join(lines)

The report caps at five segments; more than five typically exceeds a reader's working-memory budget and turns the output into noise. The guard on rank_score < 0.001 short-circuits when no segment is statistically distinguishable from the baseline, printing a clean no-regression confirmation instead of an empty table.

Confirm that when you call RegressionAnalyzer(baseline, candidate).analyze() against a result set where one domain regresses deliberately, that domain's (domain, class, length_bucket) tuple appears first in the returned list with a negative delta and a p_value below 0.05.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do include all three axes — domain, class, and length_bucket — in segment_axes — the lesson establishes that crossing at least three axes simultaneously is the minimum useful resolution; any single-axis cut lets the other two mask regressions that never surface in the aggregate pass rate.
2. ✓Do enforce the five-sample minimum before running _chi_square — chi-square contingency on a cell with fewer than five observations in either run produces unreliable p-values that will contaminate the ranked output; the n_b < 5 or n_c < 5 guard in analyze() is not optional bookkeeping.
3. ✓Do rank surviving cells by the composite |delta| × (1 − p) × |effect_size| score rather than by raw delta alone — this rewards segments that are simultaneously large in absolute change, statistically real, and meaningful in Cohen's h magnitude, which prevents a large-but-noisy delta from outranking a smaller, genuine regression at p < 0.01.

Don'ts

1. ✗Don't sort or filter segments by pass-rate delta without the (1 − p) and |Cohen's h| multipliers — a Δ of −0.15 with p = 0.9 is sampling noise; promoting it above a Δ of −0.08 with p = 0.003 and h = −0.52 misdirects investigation to the wrong segment.
2. ✗Don't render more than five segments in render_report — the lesson is explicit that more than five exceeds a reader's working-memory budget and degrades the CI report into noise; cap the slice at results[:top] with top defaulting to 5.
3. ✗Don't omit the rank_score < 0.001 guard at the top of render_report — without it, a run where no cell reaches statistical significance produces an empty Markdown table that CI parses as ambiguous output instead of the clean "No significant segment-level regression detected" confirmation the pipeline expects.