Profile agent performance

Introduction

When your hosted agent jumps from two seconds to eight on a routine query, the temptation is to optimise something quickly — add a cache, swap a model, parallelise a loop. Teams that guess at bottlenecks almost always target the wrong layer: they tune Python comprehensions while a single blocking LLM call silently accounts for 80% of every request. Profiling replaces that guesswork with measurement, and for agent workloads the measurement must span CPU time, memory allocation, and async I/O — because agents spend most of their wall-clock time waiting on the network, not computing.

By the end of this lesson you will be able to choose the right profiler for a given bottleneck — cProfile, line_profiler, memory_profiler, or yappi — instrument an agent run without restructuring it, and read the output to pinpoint the exact function or coroutine responsible.

Key Terminology

• Wall-clock time — the real elapsed time a user waits, including I/O waits. For agents this is the number that matters, because LLM API calls dominate latency and CPU-only timers undercount them.
• CPU time — time the process was actually executing on a core. Useful for finding hot loops in parsing or serialisation, but misleading on its own for I/O-bound agent code.
• Hot path — the chain of calls that accounts for the majority of cumulative time in a profile. Optimisation effort should land here first.
• Sampling vs deterministic profiling — sampling profilers (e.g. py-spy) periodically inspect the call stack with low overhead; deterministic profilers (cProfile, line_profiler) record every call and cost more but report exact counts. Agents in production usually warrant sampling; offline investigation warrants deterministic.
• RSS (Resident Set Size) — physical memory the process holds. memory_profiler reports per-line RSS deltas, which is how leaks in long-running agent loops surface.

Concepts

Four profiling axes, one decision

Profiling for agents splits cleanly along two questions: what resource and what granularity. The resource axis is CPU vs memory vs wall-clock I/O. The granularity axis is function-level vs line-level. The right tool drops out of that grid.

Why standard profilers mislead on async agents

cProfile attributes time to the function currently on the stack. In an async agent, the function on the stack during an await asyncio.sleep or a httpx.AsyncClient.post is not spending CPU — it's suspended. cProfile reports near-zero CPU for the call even though the user waited seconds. yappi solves this by switching its clock to wall-clock mode and tracking coroutines as first-class units, so the LLM round-trip shows up where it actually costs you (see Code Walkthrough).

Profile realistic workloads, not micro-benchmarks

A 10-token "hello" prompt profile looks nothing like a 4000-token tool-using turn. Profile with production-shaped inputs: real conversation history sizes, real tool counts, real concurrent request load. Cold runs (first invocation after process start) include import and JIT costs that warm runs do not — capture both, because cold latency dominates autoscaler scale-out behaviour while warm latency dominates steady-state UX.

Measure twice, optimise once

Every optimisation needs a before-and-after profile on the same workload. Without the after-profile you cannot tell whether a change helped, did nothing, or regressed a different code path. Save profile artefacts (.prof files) alongside the change in version control or CI so regressions are diffable.

Code Walkthrough

Now that you've seen the four profiling axes, why standard profilers mislead on async agents, why to profile realistic workloads rather than micro-benchmarks, and why to measure twice and optimise once, this walkthrough turns them into working code.

The example below wires all four profilers around a single agent run. Run each profiler in its own session — combining them distorts measurements.

Code snippetpython
1import asyncio, cProfile, pstats, re, json
2from pstats import SortKey
3
4import yappi
5from memory_profiler import profile as mem_profile
6
7# --- Line-level (line_profiler) ---
8# Run with: kernprof -l -v agent_script.py
9@profile  # noqa: F821  (kernprof injects this at runtime)
10def parse_llm_response(response: str) -> dict:
11    action_line = next(
12        (ln for ln in response.split("\n") if ln.startswith("Action:")), None
13    )
14    action = action_line.split(":", 1)[1].strip() if action_line else "none"
15    match = re.search(r"Arguments:\s*(\{.*\})", response, re.DOTALL)
16    return {"action": action, "arguments": json.loads(match.group(1)) if match else {}}
17
18# --- Memory-level (memory_profiler) ---
19@mem_profile
20def build_context(system_prompt: str, messages: list) -> str:
21    formatted = [f"[{m['role'].upper()}]: {m['content']}" for m in messages]
22    return system_prompt + "\n\n" + "\n\n".join(formatted)
23
24# --- CPU function-level (cProfile) ---
25def profile_cpu(agent, query: str) -> None:
26    pr = cProfile.Profile()
27    pr.enable()
28    asyncio.run(agent.run(query))
29    pr.disable()
30    pstats.Stats(pr).sort_stats(SortKey.CUMULATIVE).print_stats(15)
31    pstats.Stats(pr).print_callers(5)
32
33# --- Async wall-clock (yappi) ---
34async def profile_async(agent, query: str):
35    yappi.set_clock_type("wall")    # critical for I/O-bound agents
36    yappi.start()
37    try:
38        return await agent.run(query)
39    finally:
40        yappi.stop()
41        yappi.get_func_stats().print_all()
42
43if __name__ == "__main__":
44    # Run only one profiler at a time; comment out the others.
45    profile_cpu(agent, "Summarise this report.")
46    # asyncio.run(profile_async(agent, "Summarise this report."))

Reading the output: cProfile's top entries for an I/O-bound agent typically point to httpx or asyncio event-loop internals — the signal to re-run under yappi, which attributes wall-clock time to the LLM coroutine directly rather than spreading it across scheduler overhead. line_profiler on parse_llm_response usually flags the re.search and json.loads calls as the hot lines; memory_profiler on build_context shows the list comprehension and final string concatenation each duplicating the full conversation history in RSS.

To verify you have found the real bottleneck, make one targeted change to the top entry — for example, compiling the regex once outside the function — then re-run the same profiler: if the leading entry drops measurably or shifts to a different call, you have located the hot path; if the report looks unchanged, continue reading the output before optimising further.

Do's and Don'ts

Having walked through profiling for performance above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do profile with production-shaped inputs — prompt lengths, tool counts, and concurrency that match real traffic; micro-benchmarks hide the bottlenecks that actually matter.
2. ✓Do match the profiler to the resource — cProfile/line_profiler for CPU, memory_profiler for RSS growth, yappi for async wall-clock. A wrong-axis profile points at the wrong fix.
3. ✓Do save before-and-after profile artefacts — keep .prof files in CI so optimisation claims are reviewable and regressions are diffable.

Don'ts

1. ✗Don't use cProfile alone on async agents — it undercounts awaited I/O. The "hot" function will look like asyncio internals while the real cost is the LLM round-trip.
2. ✗Don't optimise without a profile — guessing at bottlenecks wastes effort on cold paths and frequently regresses the real hot path.
3. ✗Don't profile in development and ship the result — cold imports, debug logging, and dev-only middleware skew the numbers; re-profile in a production-like environment before drawing conclusions.