Monitor agent behavior for security anomalies

Introduction

When you deploy an AI agent that autonomously executes tool calls, queries databases, and calls external APIs, traditional request-level monitoring misses the real threat — because compromise reveals itself in the pattern of actions over time, not in any single request. An agent hijacked through indirect prompt injection may pass every individual request check while it exfiltrates data or escalates privileges across dozens of sequential steps. By the end of this lesson, you'll be able to build a behavioral baseline profiler that captures action frequency distributions, sequence patterns, resource access, and timing characteristics, then compute a drift score that gates the agent's permission level before damage can occur.

Key Terminology

• Behavioral baseline — A statistical model of normal agent operation built from a historical action log by build_baseline; it stores normalized action-type frequencies, bigram transition counts, and a known-resource set that together define what the agent's "normal" looks like.
• Drift score — A 0–1 composite anomaly value returned by score_drift that averages the fraction of unknown action types and the fraction of unknown resources observed in a live window; a score above 0.5 maps to Level 2 (rate-limited) or higher containment.
• Action bigram — A consecutive pair of action types (for example, ("db_query", "llm_call")) recorded during baseline construction; the bigram transition table captures which action-to-action sequences appear in normal operation, so novel pairs in live traffic signal a behavioral anomaly.
• Known-resource set — The set of resource identifiers (database names, API endpoints, file paths) extracted from the historical action log by build_baseline; any resource accessed in live operation that is absent from this set increments the unknown-resource signal in score_drift.
• Indirect prompt injection — An attack where malicious instructions embedded in tool outputs (API responses, database records, web content) redirect the agent's subsequent actions; it is the primary threat driving behavioral monitoring because each individual request can appear valid while the session-level action pattern reveals the compromise.
• Permission level — One of five containment tiers (Level 0 through Level 4) that the monitoring pipeline assigns based on the drift score, ranging from full permissions at Level 0 to full agent isolation at Level 4.

Concepts

Why Request-Level Checks Are Not Enough

A traditional security boundary inspects each request in isolation: is this tool call authorized? Is this API parameter valid? These checks are necessary but insufficient for autonomous agents. An agent compromised through indirect prompt injection continues to make individually authorized calls — it queries the same database it always has, calls the same LLM endpoint it always has — but the sequence of those calls shifts: database queries suddenly precede external POST requests that never co-occurred before; a resource the agent has never legitimately needed appears in the access log. No single call triggers an alert, yet across dozens of sequential steps the agent can exfiltrate data or escalate privileges. The core insight this lesson builds on is that compromise reveals itself in the pattern of actions over time, not in any individual action.

Four Dimensions That Together Define "Normal"

A behavioral baseline captures four complementary dimensions, each sensitive to a different attack pattern. Action-type frequency detects categorical shifts — a sudden spike in file-write operations or external API calls relative to the historical distribution. Bigram sequences detect novel action chains: a ("db_query", "external_post") bigram that never appeared in training data is a stronger signal than either action alone, because indirect prompt injection typically rewires the agent's control flow into sequences that are locally plausible but globally unprecedented. Resource access catches lateral movement the moment the agent touches a database, endpoint, or file it has never legitimately needed — a binary violation that fires regardless of how normal the action-type distribution looks. Timing characteristics detect pacing anomalies: automated exploitation typically drives inter-action gaps far below the historical mean, while an adversary manually steering the agent through injected instructions produces gaps far above it. All four dimensions matter because an attacker can evade any single one (for example, maintaining a plausible action-type distribution while accessing a novel resource), but evading all four simultaneously is substantially harder.

The Drift Score as a Graduated Permission Gate

Rather than triggering a binary alarm, score_drift returns a continuous 0–1 value that feeds a five-level permission gate (see Code Walkthrough). The score averages two independent anomaly signals — the fraction of live action types absent from baseline["frequency"] and the fraction of live resources absent from baseline["known_resources"]. Averaging independent signals provides robustness: one signal drifting slightly (for example, a new but benign action type introduced by a legitimate feature update) does not immediately push the composite into a high-containment tier. But correlated drift across both signals — the hallmark of a genuine compromise — escalates the score quickly. The score's continuous nature also matters architecturally: the same value drives five graduated responses rather than a single on/off alarm, which reduces both false-positive disruption and the blast radius of actual incidents. A key normalization check ties the whole system together — replaying the same log used to build the baseline through score_drift should return a score below 0.1. A higher value on the training data itself points to a bug in build_baseline (typically a missed normalization step) rather than real behavioral change, and catching that early prevents the drift detector from crying wolf on every live window.

Code Walkthrough

Now that you understand the four behavioral dimensions — action frequency, sequence patterns, resource access, and timing — the next step is seeing how a baseline profiler captures those dimensions from a live stream of agent action events.

The flowchart below shows the full monitoring pipeline: raw agent actions feed a baseline profiler, a drift detector computes a composite anomaly score, and that score determines which permission level the agent operates under.

The Python snippet below implements a minimal baseline profiler covering the three most portable dimensions — frequency distribution, bigram (2-gram) action sequences, and known-resource access — while leaving timing characteristics as a straightforward extension that follows the same pattern.

Code snippetpython
1from collections import Counter
2
3def build_baseline(action_log: list[dict]) -> dict:
4    """Build a statistical baseline from historical agent action events.
5
6    Each event must have an 'action_type' key and an optional 'resource' key.
7    Returns frequency distribution, bigram transition counts, and known resources.
8    """
9    action_types = [e["action_type"] for e in action_log]
10    total = len(action_types) or 1
11
12    # Action type frequency distribution
13    freq = {k: v / total for k, v in Counter(action_types).items()}
14
15    # Bigram (2-gram) sequence pattern counts
16    bigrams = Counter(
17        (action_types[i], action_types[i + 1])
18        for i in range(len(action_types) - 1)
19    )
20
21    # Resource access allowlist derived from history
22    known_resources = {e["resource"] for e in action_log if e.get("resource")}
23
24    return {
25        "frequency": freq,
26        "bigrams": dict(bigrams),
27        "known_resources": known_resources,
28    }
29
30def score_drift(baseline: dict, live_actions: list[dict]) -> float:
31    """Return a 0–1 drift score comparing live actions against the baseline.
32
33    A score above 0.5 maps to Level 2 (rate-limited) or higher.
34    """
35    if not live_actions:
36        return 0.0
37
38    action_types = [e["action_type"] for e in live_actions]
39    total = len(action_types)
40
41    # Fraction of action types absent from the baseline
42    unknown_type_rate = sum(
43        1 for a in action_types if a not in baseline["frequency"]
44    ) / total
45
46    # Fraction of accessed resources not in the known-resource set
47    live_resources = {e["resource"] for e in live_actions if e.get("resource")}
48    unknown_resource_rate = len(live_resources - baseline["known_resources"]) / max(
49        len(live_resources), 1
50    )
51
52    return (unknown_type_rate + unknown_resource_rate) / 2

build_baseline ingests a historical action log and returns the baseline: normalized type frequencies, bigram transition counts (which expose novel action sequences not present in normal operation), and a known-resource set (which catches a compromised agent reaching databases or APIs it has never legitimately touched). score_drift then computes a composite 0–1 score by averaging two anomaly signals — unknown action types and unknown resources — against a live window of activity. Extending this to capture timing characteristics follows the same structure: record inter-action time deltas during profiling, then flag windows where gaps fall outside the historical mean by more than two standard deviations.

Confirm that score_drift returns a value below 0.1 on a replay of the same action log used to build the baseline — a non-zero score against its own training data indicates a normalization bug in build_baseline.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do validate score_drift by replaying the exact log used to build the baseline and confirming the result is below 0.1 — a non-zero score against its own training data exposes a normalization bug in build_baseline (e.g., a mismatch between how frequency stores relative counts and how the unknown-type lookup works) before it silently inflates scores on clean production sessions.
2. ✓Do capture bigram (2-gram) transition counts alongside action-type frequencies in build_baseline — frequency distributions alone cannot distinguish a compromised agent that reuses legitimate action types in a novel attack sequence; bigrams expose transitions such as query_db → external_call that never appeared in normal operation and would score zero anomaly under a frequency-only baseline.
3. ✓Do add timing characteristics as a third drift signal by following the same two-signal averaging pattern already used in score_drift — record inter-action time deltas during baseline profiling and flag live windows where gaps fall outside the historical mean by more than two standard deviations, keeping the composite score architecture consistent and extensible without restructuring the detector.

Don'ts

1. ✗Don't substitute per-request checks for score_drift as the primary gate against a compromised agent — indirect prompt injection reveals itself in the pattern of actions over time, not in any single request; an agent exfiltrating data across dozens of sequential steps will pass every individual request check while the drift score steadily climbs toward Level 3 (Restricted) or Level 4 (Isolated).
2. ✗Don't include known-anomalous sessions when building the baseline — contaminated logs fold attack-introduced action types and novel resources directly into frequency and known_resources, permanently blunting score_drift's sensitivity to those exact threat patterns on future sessions.
3. ✗Don't collapse the five permission levels into a single binary threshold on the drift score — score_drift returns a continuous 0–1 value specifically to support graduated escalation from Level 0 (Full Permissions) through Level 4 (Isolated); a single cutoff discards the rate-limiting and elevated-monitoring bands that provide proportionate responses before a score reaches the emergency threshold.