Validate orchestration correctness with agent trajectory evaluation

Introduction

When your multi-agent orchestrator returns the right final answer, it's tempting to mark the run green — but the path it took may have been a routing accident, a costly loop, or a privileged tool fired without authorisation. Teams that score only on outcomes are blind to all of that, and the failures surface later as runaway token bills, silent regressions, and security incidents. By the end of this lesson you'll be able to evaluate agent trajectories with a four-axis LLM-as-judge rubric, anchor scoring against golden runs, and wire trajectory verdicts into both CI gates and production sampling.

Key Terminology

• Trajectory: the ordered, timestamped record of every agent invocation, tool call, handoff, and token cost an orchestrator emitted while producing a final answer — the substrate trajectory evaluation reads.
• Four-axis rubric: the scoring frame applied by the LLM-as-judge — efficiency, correctness, and economy on 1–5 scales plus safety as a binary pass/fail — kept small to prevent judge drift.
• Golden trajectory: a hand-curated minimum-step, correctly-routed, properly-authorised reference run for a canonical task, stored under evals/trajectories/ and passed to the judge as the anchor against which a live run is compared.
• Structural check: a deterministic, token-free pre-judge pass that flags repeated identical tool calls and unauthorised tool invocations directly from the trajectory before any LLM scoring runs.
• Trajectory verdict: the structured output of the evaluator (per-axis scores, safety flag, structural flags, rationale) consumed by CI gates and production-sampling pipelines to block releases or page on regressions.

Concepts

Why path matters more than answer

Three failure modes that outcome scoring misses:

1. Wrong agent, lucky answer. The supervisor sent a SQL question to the web-search agent, which happened to find a blog post quoting the right number. The answer is right; the routing is broken.
2. Loops and over-delegation. The supervisor calls the planner three times in a row to "refine" the same plan. Final answer is fine; token cost is 4x what it should be.
3. Unauthorised tool use. A tool with write access to production was invoked by an agent that should be read-only. The answer doesn't reveal it; only the trajectory does.

A trajectory rubric for LLM-as-judge

The rubric below has four axes. Keep them few and orthogonal — judges drift when asked to score on twelve dimensions.

Safety is binary on purpose: a single unauthorised privileged-tool call is a release blocker, not a 3-out-of-5.

Golden trajectories as anchors

LLM judges scoring in the abstract are noisy. Anchor them by hand-curating one to three "ideal" trajectories per canonical task — for example, "answer a question that requires both retrieval and code execution." A golden shows the minimum-step, correctly-routed, properly-authorised path. The judge prompt becomes "compare this run to the golden below and score on the four-axis rubric," which is dramatically more reliable than free-form scoring. Store goldens alongside the eval set under evals/trajectories/.

Operating discipline

• Keep the rubric to four axes or fewer; judges drift when overloaded.
• Pair every judgement with a golden when one exists; free-form scoring is too noisy for SLOs.
• Run structural checks before the LLM judge — they're free and they catch the worst failures deterministically.
• Version goldens. When the orchestrator graph legitimately changes, goldens must change with it; stale goldens silently make every run look bad.
• Sample production at a rate you can actually pay for. Trajectory eval is not free — budget tokens explicitly.

Pitfalls

• Treating the judge as ground truth. The judge is a noisy estimator; calibrate it against human review on a held-out set quarterly.
• Over-fitting to the golden. If goldens are too prescriptive, novel-but-correct paths get scored low. Keep goldens minimal and let the rubric carry semantic judgement.
• Scoring outcome inside trajectory eval. Keep them separate. Trajectory eval answers "was the path good?", outcome eval answers "was the answer right?". Mixing them hides which dimension regressed.
• Sampling only successful runs. Sample uniformly, including refusals and errors — that's where path pathologies hide.
• No drift detection on the judge. Pin the judge model and prompt; when you change either, re-baseline all thresholds.

Code Walkthrough

What is a trajectory

A trajectory is the ordered, timestamped record of every step the orchestrator took to produce a final answer. It includes:

• Each agent invocation (which agent, which prompt, which model).
• Each tool call (name, arguments, latency, return payload).
• Each intermediate message — supervisor handoffs, scratchpad reasoning, retry decisions.
• Token and cost accounting per step.

If the chapter's tracing layer is doing its job, the trajectory is already on disk as a tree of OpenTelemetry spans or a structured JSON log. Trajectory evaluation is the layer that reads that record and asks: was this a good run?

The TrajectoryEvaluator

The evaluator loads a recorded trajectory, optionally pairs it with a golden, runs the LLM judge, and emits a structured verdict that downstream systems can act on. It separates structural checks (cheap, deterministic) from the LLM judgement (expensive, noisy) — structural checks catch obvious failures before any tokens are spent.

Code snippetpython
1from dataclasses import dataclass, field
2from pathlib import Path
3import json
4
5@dataclass
6class TrajectoryStep:
7    kind: str          # "agent_invoke" | "tool_call" | "handoff"
8    actor: str         # supervisor / retrieval_agent / etc
9    name: str          # tool name or agent name
10    args: dict
11    output: str
12    tokens: int
13    latency_ms: int
14
15@dataclass
16class Trajectory:
17    task_id: str
18    query: str
19    steps: list[TrajectoryStep]
20    final_answer: str
21    total_tokens: int = 0
22    total_latency_ms: int = 0
23
24    @classmethod
25    def from_json(cls, path: Path) -> "Trajectory":
26        raw = json.loads(Path(path).read_text())
27        steps = [TrajectoryStep(**s) for s in raw["steps"]]
28        return cls(
29            task_id=raw["task_id"],
30            query=raw["query"],
31            steps=steps,
32            final_answer=raw["final_answer"],
33            total_tokens=sum(s.tokens for s in steps),
34            total_latency_ms=sum(s.latency_ms for s in steps),
35        )
36
37@dataclass
38class Verdict:
39    efficiency: int
40    correctness: int
41    economy: int
42    safety_pass: bool
43    rationale: str
44    structural_flags: list[str] = field(default_factory=list)
45
46class TrajectoryEvaluator:
47    def __init__(self, judge_client, allowed_tools_by_agent: dict[str, set[str]]):
48        self.judge = judge_client
49        self.allowed = allowed_tools_by_agent
50
51    def structural_checks(self, traj: Trajectory) -> list[str]:
52        flags: list[str] = []
53        # repeated identical tool calls
54        seen: dict[tuple, int] = {}
55        for s in traj.steps:
56            if s.kind == "tool_call":
57                key = (s.actor, s.name, json.dumps(s.args, sort_keys=True))
58                seen[key] = seen.get(key, 0) + 1
59        for key, n in seen.items():
60            if n >= 3:
61                flags.append(f"repeated_tool_call:{key[1]}:{n}")
62        # unauthorised tool calls
63        for s in traj.steps:
64            if s.kind == "tool_call":
65                allowed = self.allowed.get(s.actor, set())
66                if s.name not in allowed:
67                    flags.append(f"unauthorised_tool:{s.actor}->{s.name}")
68        return flags
69
70    def evaluate(self, traj: Trajectory, golden: Trajectory | None = None) -> Verdict:
71        flags = self.structural_checks(traj)
72        prompt = self._build_judge_prompt(traj, golden, flags)
73        scored = self.judge.score(prompt)  # returns dict matching Verdict
74        return Verdict(
75            efficiency=scored["efficiency"],
76            correctness=scored["correctness"],
77            economy=scored["economy"],
78            safety_pass=scored["safety_pass"] and not any(
79                f.startswith("unauthorised_tool") for f in flags
80            ),
81            rationale=scored["rationale"],
82            structural_flags=flags,
83        )

The judge prompt should ask for a JSON object matching Verdict, include the rubric verbatim, and inline both the run trajectory and the golden so the model can compare side by side.

Comparing live vs golden trajectories

For sampled production runs you usually don't have a hand-curated golden, but you do have a corpus of past well-scored runs for similar tasks. Two cheap heuristics narrow the search before the LLM judge runs:

• Edit distance over the step sequence. Treat each step as a token (supervisor:delegate:retrieval, retrieval:tool_call:vector_search, etc.). Levenshtein distance against each candidate golden gives a structural similarity score.
• KL divergence over step-kind histograms. Compares how often each kind of step occurs without caring about order — useful for flagging "this run did 8x as many tool calls as the golden."

Code snippetpython
1def step_signature(step: TrajectoryStep) -> str:
2    return f"{step.actor}:{step.kind}:{step.name}"
3
4def edit_distance(a: list[str], b: list[str]) -> int:
5    # standard Levenshtein, omitted for brevity
6    ...
7
8def nearest_golden(traj: Trajectory, goldens: list[Trajectory]) -> Trajectory:
9    sig = [step_signature(s) for s in traj.steps]
10    return min(goldens, key=lambda g: edit_distance(sig, [step_signature(s) for s in g.steps]))

The LLM judge then runs on the (run, nearest-golden) pair and produces a semantic verdict that the heuristics alone cannot — for example, recognising that two structurally different paths are equally valid for an open-ended research question.

Wiring into CI and production

Trajectory evaluation has two homes:

1. CI gate. A fixed eval set of fifty to two hundred canonical tasks runs on every change to the orchestrator graph, agent prompts, or tool definitions. Each task has a golden. The CI job fails if the median efficiency or correctness score drops below threshold, or if any safety check fails. This catches regressions before they reach production.
2. Production sampler. One to five percent of production runs are sampled, paired with the nearest golden, and scored asynchronously by the judge. Aggregated scores are emitted as metrics: trajectory_efficiency, trajectory_correctness, trajectory_economy_p50, trajectory_safety_failures_total. Alerts fire on regressions.

Do's and Don'ts

Do's

1. ✓Do run structural_checks before invoking the LLM judge — flagging repeated_tool_call (≥3 identical (actor, name, args) triples) and unauthorised_tool violations deterministically means obviously-broken runs are caught without spending any judge tokens, keeping evaluation cost proportional to run quality.
2. ✓Do anchor safety_pass to both the judge's verdict and the unauthorised_tool structural flags — the scored["safety_pass"] and not any(f.startswith("unauthorised_tool") for f in flags) override in evaluate() ensures that an agent firing a tool outside its allowed_tools_by_agent scope can never receive a green safety verdict, even when the final answer is correct.
3. ✓Do use nearest_golden() with edit distance over step_signature strings (actor:kind:name) before the LLM judge runs — pairing a live trajectory against the structurally closest golden prevents rubric axes like efficiency and economy from being inflated or deflated by a mismatched reference; KL divergence over step-kind histograms adds a cheap second filter for frequency anomalies the edit distance misses.

Don'ts

1. ✗Don't score orchestrator runs on final-answer correctness alone — a run that reaches the right answer via a routing accident, an unbounded retry loop, or an unauthorized code_executor call is indistinguishable from a correct run at the output layer, and those failures surface later as runaway token bills, silent CI regressions, and security incidents.
2. ✗Don't remove the structural AND from safety_pass in Verdict — if unauthorised_tool flags are not independently ANDed into the safety verdict, a high judge score on the efficiency or correctness axes can mask an agent that called a privileged tool it was never granted access to, defeating the entire allowed_tools_by_agent enforcement boundary.
3. ✗Don't pass raw TrajectoryStep objects to edit-distance without first converting them to step_signature strings — treating unaggregated step objects as comparison tokens makes every step look structurally unique, so nearest_golden() returns an arbitrary match and the judge scores the live run against an unrelated reference trajectory.