Build agent benchmarks with task suites

Introduction

When you ship an agent behind a single "task success" number, you hide the two failure modes that bite hardest in production: an agent that finishes in triple the steps, and an agent that crumbles the moment a tool returns an error. Teams that score only completion miss regressions where a prompt change quietly doubles tool calls or strips out retry logic — dashboards stay green until the bill arrives or a user complains. By the end of this lesson you will be able to design a three-dimensional benchmark suite that scores completion, efficiency, and recovery independently, compute regression thresholds from baseline variance, and emit a PASS/FAIL verdict that CI can gate on.

Key Terminology

• Completion rate — fraction of benchmark tasks where the agent reaches the correct terminal state; the headline correctness metric and the dimension every team measures first.
• Efficiency ratio — optimal steps divided by actual steps for completed tasks; surfaces wasted tool calls and planning drift that pure completion hides.
• Recovery rate — fraction of tasks where the agent encountered a tool error mid-trajectory and still completed; isolates resilience from baseline correctness.
• Regression threshold — baseline_mean − 2·baseline_std per metric; the line below which a run is statistically distinguishable from the baseline distribution.
• Deterministic tool mock — a fake tool that returns fixed outputs (and fixed injected failures) so score variance reflects agent changes, not external service jitter.

Concepts

Three independent quality dimensions

A single completion-rate number collapses correctness, cost, and resilience, so a five-point drop could be any of the three — or two of them cancelling out. Score the dimensions separately: completion answers "did it finish?", efficiency answers "at what cost?", recovery answers "what happens when tools fail?". Each dimension has a distinct fix path — a completion regression points at the tool integration or prompt, an efficiency regression points at planning, a recovery regression points at retry logic (see Code Walkthrough).

Deterministic mocks over live tools

Live tools introduce variance the benchmark cannot attribute. A flaky search API can produce a 3-point completion swing run-to-run that looks identical to a real regression. Replace every external tool with a deterministic mock that returns fixed payloads — and, for recovery tasks, fails in fixed ways at fixed steps. The benchmark's job is to detect agent changes; anything else producing variance is noise that has to go.

Regressions from baseline variance, not magic numbers

A static gate like "completion must exceed 0.90" either fires on harmless noise or misses real drops, depending on how lucky you got picking the number. Compute thresholds from observed variance instead: run the baseline N times, store mean and standard deviation per metric, and flag any value below mean − 2·std. That line is roughly the 95th percentile of normal run-to-run drift — anything below is statistically a real change worth investigating.

Code Walkthrough

Building on the three dimensions and the variance-based threshold from the previous section, the harness below ties them together: it captures per-task results in a fixed schema, scores all three dimensions in one pass, and applies the mean − 2·std regression check to emit a CI-gateable verdict.

Code snippetpython
1import json
2import statistics
3from dataclasses import dataclass
4
5@dataclass
6class BenchmarkResult:
7    task_id: str
8    completed: bool
9    actual_steps: int
10    optimal_steps: int
11    recovered_from_error: bool
12
13def score_benchmark(results: list[BenchmarkResult], baselines: dict) -> dict:
14    completion_rate = sum(r.completed for r in results) / len(results)
15    efficiencies = [r.optimal_steps / r.actual_steps for r in results if r.completed]
16    efficiency_mean = statistics.mean(efficiencies) if efficiencies else 0.0
17    error_tasks = sum(1 for r in results if r.actual_steps > r.optimal_steps)
18    recovery_rate = sum(r.recovered_from_error for r in results) / max(error_tasks, 1)
19    scores = {
20        "completion_rate": round(completion_rate, 3),
21        "efficiency_mean": round(efficiency_mean, 3),
22        "recovery_rate": round(recovery_rate, 3),
23    }
24    regressions = []
25    for metric, value in scores.items():
26        base = baselines.get(metric, {})
27        threshold = base.get("mean", 0) - 2 * base.get("std", 0.05)
28        if value < threshold:
29            regressions.append(f"{metric}: {value:.3f} < threshold {threshold:.3f}")
30    scores["regressions"] = regressions
31    scores["verdict"] = "PASS" if not regressions else "FAIL"
32    return scores
33
34baselines = {
35    "completion_rate": {"mean": 0.93, "std": 0.03},
36    "efficiency_mean": {"mean": 0.85, "std": 0.04},
37    "recovery_rate": {"mean": 0.78, "std": 0.06},
38}
39results = [
40    BenchmarkResult("task_1", True, 5, 5, False),
41    BenchmarkResult("task_2", True, 8, 5, True),
42    BenchmarkResult("task_3", False, 12, 4, False),
43]
44print(json.dumps(score_benchmark(results, baselines), indent=2))

BenchmarkResult is the fixed per-task schema — completion is boolean, efficiency derives from the actual_steps/optimal_steps pair, and recovery is a boolean conditioned on whether the task hit an injected error. score_benchmark computes the three dimension scores in one pass, then applies the mean − 2·std threshold per metric to flag regressions. The verdict is PASS only when zero metrics regress, which is the signal CI branches on. You will know it works when running against the stored baseline produces "verdict": "PASS" with an empty regressions list, and intentionally degrading the agent (e.g. doubling step counts) flips it to FAIL with efficiency_mean named in the regression list.

Do's and Don'ts

Having just walked through the harness, distil the operational lessons into the rules below — what to lock in across every benchmark run, and what to refuse no matter how tempting the shortcut.

Do's

1. ✓Do score completion, efficiency, and recovery as separate metrics — a single composite score hides which dimension regressed and routes investigation to the wrong place.
2. ✓Do compute regression thresholds from observed baseline variance — mean − 2·std adapts to each metric's natural noise instead of firing on harmless jitter or missing real drops.
3. ✓Do mock every external tool deterministically inside the benchmark — live-tool variance is indistinguishable from agent regressions and will erode trust in the suite within weeks.

Don'ts

1. ✗Don't gate CI on raw metric values — a hard floor like "completion ≥ 0.90" goes stale the moment the baseline shifts; gate on the regression verdict instead.
2. ✗Don't store baselines outside version control — a baseline that drifts silently is worse than no baseline, because the verdict still reads PASS.
3. ✗Don't fold recovery into completion — a task that succeeded after a retry is operationally different from one that succeeded first try; collapsing them masks resilience regressions.