Score agent tool selection with DeepEval 3.0 and Vertex AI Agent Evaluation

Introduction

When you evaluate a multi-step agent only on its final answer, you cannot tell whether it picked the right tool, recovered from a failed reasoning step, or got lucky on a path that won't generalize — a bad tool choice, a skipped retrieval, and a flawed planning step all collapse into one opaque score. In this lesson you'll use DeepEval's @observe decorator to instrument each function in a multi-step agent so every LLM call, tool invocation, and synthesis step emits a structured span. By the end you'll be able to attach the captured trace to an LLMTestCase and run trajectory-aware metrics (ToolCorrectnessMetric, AgentGoalAccuracyMetric) that score individual segments of the run, turning agent evaluation into a per-step diagnostic instead of a single pass/fail verdict.

Key Terminology

• @observe decorator: DeepEval function wrapper (type="llm", type="tool", type="retriever", etc.) that emits a structured span for each call, recording input, output, latency, and parent–child relationships in the active trace.
• Trace span: A single recorded execution unit inside a trace tree, produced by an @observe-wrapped function and carrying metadata that component-level metrics consume.
• TraceManager context: Context manager that opens a root trace, captures every nested @observe span emitted inside its with block, and returns the assembled hierarchical trace object.
• ToolCorrectnessMetric: Trajectory metric that compares tools the agent actually invoked (from the trace) against expected_tools on the test case, reporting precision and recall.
• AgentGoalAccuracyMetric: Reference-free judge metric that scores whether the agent's final output satisfies the user's stated goal, suitable for open-ended tasks with multiple correct answers.

Concepts

@observe-based trajectory evaluation rests on three ideas working together. First, decoration is the instrumentation contract: wrapping a function with @observe(type=..., name=...) is what makes it visible to DeepEval's trace pipeline — untagged functions stay invisible, so coverage is a deliberate authoring choice rather than automatic. Second, the trace tree is the evaluation substrate: spans nest under whichever decorated function is currently on the call stack, so wrapping run_agent callees inside a TraceManager block produces a hierarchical record of planning → tool use → synthesis that mirrors actual execution order. Third, metrics operate on segments, not just outputs: ToolCorrectnessMetric reads the tool spans out of the trace and checks them against expected_tools; AgentGoalAccuracyMetric reads the goal and the final answer and asks an LLM judge whether the trajectory satisfied intent. Because both metrics live on the same LLMTestCase, a single evaluate(...) call returns separate scores and reasons for tool selection and goal achievement — letting you tell apart "agent picked the wrong tool" from "agent picked the right tool but synthesized a bad answer" (see Code Walkthrough).

Code Walkthrough

Building on the three ideas just covered, the snippet below demonstrates them together: @observe decorators tag each agent step as an llm or tool span, the TraceManager block collects nested spans into one hierarchical trace, and evaluate runs the two component-level metrics against that trace.

Code snippetpython
1from deepeval import evaluate
2from deepeval.tracing import observe, TraceManager
3from deepeval.metrics import ToolCorrectnessMetric, AgentGoalAccuracyMetric
4from deepeval.test_case import LLMTestCase
5
6@observe(type="llm", name="planner")
7def plan_next_action(query: str) -> str:
8    response = llm_client.chat(messages=[{"role": "user", "content": query}])
9    return response.content
10
11@observe(type="tool", name="search_api")
12def search_documents(query: str) -> list[dict]:
13    return vector_store.similarity_search(query, k=5)
14
15@observe(type="llm", name="synthesizer")
16def synthesize_answer(context: list[dict], query: str) -> str:
17    prompt = f"Context: {context}\nQuestion: {query}"
18    return llm_client.chat(messages=[{"role": "user", "content": prompt}]).content
19
20def run_agent(user_query: str):
21    with TraceManager() as trace:
22        action = plan_next_action(user_query)
23        docs = search_documents(action)
24        answer = synthesize_answer(docs, user_query)
25    return answer, trace
26
27answer, trace = run_agent("What is our refund policy?")
28test_case = LLMTestCase(
29    input="What is our refund policy?",
30    actual_output=answer,
31    expected_tools=["search_api"],
32    trace=trace,
33)
34results = evaluate(
35    test_cases=[test_case],
36    metrics=[ToolCorrectnessMetric(threshold=0.8), AgentGoalAccuracyMetric()],
37)

The three @observe-decorated functions emit one span each per call; nesting is automatic because every call happens inside the TraceManager context, so the resulting trace mirrors planner → tool → synthesizer order. LLMTestCase ties that trace to expected_tools=["search_api"], and evaluate returns one score per metric with a reason field explaining the judgment. When ToolCorrectnessMetric reports low precision the agent called a tool not in expected_tools; low recall means it skipped a required tool. AgentGoalAccuracyMetric returns a reference-free judgment on whether the final answer satisfies the input. You'll know it works when the result contains both metrics scored with non-empty reason strings.

Do's and Don'ts

Do's

1. ✓Do decorate every agent function with @observe(type=..., name=...) — set type="llm" for plan_next_action/synthesize_answer and type="tool" for search_documents so ToolCorrectnessMetric can correctly separate tool spans from LLM spans; using the wrong type causes the metric to misclassify calls and report misleading precision/recall scores.
2. ✓Do wrap the full agent run in a TraceManager context and pass the returned trace object to LLMTestCase(trace=trace) — spans emitted by @observe are only collected into a hierarchical trace while a TraceManager is active, so calling plan_next_action, search_documents, or synthesize_answer outside that block silently discards the spans and leaves evaluate with nothing to score.
3. ✓Do populate expected_tools with the exact strings passed to @observe(name=...) — ToolCorrectnessMetric matches tool span names against expected_tools by string equality, so listing "search_api" while the decorator reads name="search_documents" counts as both a missed required tool and an unexpected invocation, zeroing precision and recall simultaneously.

Don'ts

1. ✗Don't evaluate a multi-step agent using only output-level metrics like exact-match against the final answer — doing so collapses planner failures, wrong tool choices, and skipped retrievals into a single opaque score, making the per-step span data emitted by @observe invisible to any metric and defeating the purpose of trajectory instrumentation.
2. ✗Don't rename a tool's @observe(name=...) string without updating expected_tools in every LLMTestCase that references it — ToolCorrectnessMetric treats the old name as a missing expected tool and the new name as an unexpected invocation, so a cosmetic rename silently produces a zero precision-recall score with no error message to alert you.
3. ✗Don't apply @observe to helper functions that aggregate or orchestrate the decorated steps (like run_agent itself) without also keeping the TraceManager inside it — nesting span emission outside an active TraceManager context drops those spans, and the trace attached to LLMTestCase will be incomplete, causing AgentGoalAccuracyMetric to judge the run on a truncated trajectory missing the planner or synthesizer steps.