Monitor injection attempts with Prometheus and Grafana

Introduction

Engineers often instrument injection defenses without any visibility into whether those defenses are actually working—leaving security teams blind to attack frequency, which vectors are most active, and whether guards are catching or missing attempts in production. Prometheus and Grafana close that gap by turning raw guard-chain events into quantified metrics and visual dashboards organized around operational awareness, threat intelligence, and defense effectiveness. By the end of this lesson, you'll be able to wire Prometheus counters, histograms, and gauges into a guard pipeline, configure threshold-based alert rules that notify the security team during coordinated attacks, and interpret the resulting Grafana security dashboard to make data-driven defense-tuning decisions.

Key Terminology

• Prometheus Counter — a monotonically increasing metric that records cumulative totals; used in this lesson as SCAN_REQUESTS (labeled by endpoint and result) and INJECTION_DETECTIONS (labeled by guard_type, vector, and severity) to track scan volumes and detection counts over time.
• Prometheus Histogram — a metric that samples observed values into configurable bucket boundaries and exposes count, sum, and per-bucket totals; GUARD_LATENCY uses custom buckets from 1 ms to 1 s so that both fast pattern-match guards and slower LLM-as-judge calls appear in the same distribution.
• Prometheus Gauge — a metric whose value can rise or fall at any point in time, representing current state rather than accumulated totals; ACTIVE_GUARDS uses a gauge to confirm how many guards are loaded after a configuration change.
• Metric Label — a key-value dimension attached to a Prometheus metric at definition time (e.g., ["guard_type", "vector", "severity"] on INJECTION_DETECTIONS) that lets Grafana panels slice, filter, and aggregate data along each dimension independently.
• Alert Rule — a Prometheus YAML stanza pairing a PromQL threshold expression with a for duration clause; the HighInjectionRate rule fires only after the five-minute detection rate exceeds 50 per minute for two consecutive minutes, preventing false alerts from brief traffic spikes.
• Defense-tuning cycle — the iterative loop in which histogram latency data and detection-count data are compared to justify raising or lowering a guard's priority; for example, if GUARD_LATENCY shows an LLM-as-judge consistently costs 300 ms but INJECTION_DETECTIONS shows it catches only 5 % of attempts beyond the pattern guard, that data justifies raising the LLM judge's short-circuit threshold.

Concepts

The Observability Gap in Injection Defense

Deploying a guard chain without instrumentation creates a critical blind spot: the defenses may be running, but you have no way to know whether they are catching attacks, how often those attacks arrive, or which vectors are most active. Prometheus and Grafana close this gap by treating the guard pipeline as a source of structured telemetry. Raw guard-chain events — a scan request, a detection, a latency sample — become counters, histograms, and gauges that can be queried, visualized, and alerted on in real time.

Security monitoring for an injection guard serves three distinct purposes. Operational awareness answers: are all guards loaded and healthy? Threat intelligence answers: what attack vectors are arriving, at what rate, and at what severity? Defense effectiveness answers: what fraction of requests are being blocked, and is the false positive rate within acceptable bounds? These three purposes map directly to the Grafana dashboard rows (see Code Walkthrough), so every metric you define should be traceable to at least one of these questions.

Choosing the Right Metric Type

Prometheus exposes three primitive types, and choosing the wrong one for a given signal wastes label cardinality or loses information. Counters are appropriate for anything that only grows: total scan requests and total detections both fit this shape. Histograms are appropriate for latency: they preserve the full distribution so you can query the 50th, 95th, and 99th percentile of guard_evaluation_duration_seconds — not just the average — and the custom bucket boundaries (1 ms through 1 s) span both fast regex guards and slow LLM-as-judge calls. Gauges are appropriate for instantaneous state: ACTIVE_GUARDS fluctuates when the guard chain reloads, and a gauge reflects the current value rather than accumulating noise.

Labels amplify all three types. The three labels on INJECTION_DETECTIONS — guard_type, vector, and severity — mean a single counter definition powers the attack-analysis row's time-series graph (grouped by vector), its severity pie chart, and a per-guard detection breakdown, with no additional metric registrations required.

Alerts and the Data-Driven Tuning Loop

Alert rules translate quantified thresholds into operational responses. The for duration clause is the key design choice: without it, a single-second spike in guard_injection_detections_total would page the on-call team; with for: 2m, the rule requires the rate to stay elevated across two consecutive evaluation windows before firing. This is the difference between an alert that signals a coordinated attack and one that triggers on a momentary anomaly.

The same metrics that drive alerts also drive guard reconfiguration. If histogram data shows an LLM-as-judge guard consistently adding latency while detection counters show it contributes only marginally beyond what earlier guards caught, the case for raising its short-circuit threshold is grounded in numbers rather than intuition (see Code Walkthrough). This feedback loop — measure cost, measure incremental catch rate, adjust threshold, re-measure — is what turns a one-time deployment into a continuously improving defense posture.

Code Walkthrough

Now that you understand why security monitoring matters and how the Grafana dashboard rows surface threat intelligence and guard performance, let's wire up the Prometheus instrumentation that feeds all four of those panels.

The guard chain exposes four metric types. Counters accumulate scan request totals and detection counts. Histograms capture latency distributions for the full guard chain and each individual guard. Gauges track current state—how many guards are active and what the estimated false positive rate is. The code below defines one of each:

Code snippetpython
1from prometheus_client import Counter, Histogram, Gauge
2
3SCAN_REQUESTS = Counter(
4    "guard_scan_requests_total",
5    "Total number of scan requests processed",
6    ["endpoint", "result"]  # result: "allowed" or "blocked"
7)
8
9GUARD_LATENCY = Histogram(
10    "guard_evaluation_duration_seconds",
11    "Time spent evaluating each guard",
12    ["guard_type"],
13    buckets=[0.001, 0.005, 0.01, 0.05, 0.1, 0.25, 0.5, 1.0]
14)
15
16INJECTION_DETECTIONS = Counter(
17    "guard_injection_detections_total",
18    "Injection attempts detected by guard type and vector",
19    ["guard_type", "vector", "severity"]
20)
21
22ACTIVE_GUARDS = Gauge(
23    "guard_active_guards_count",
24    "Number of active guards in the chain"
25)

SCAN_REQUESTS is labeled by endpoint and result so you can filter by allowed-versus-blocked traffic at each entry point. GUARD_LATENCY uses custom bucket boundaries from 1 ms to 1 s—wide enough to capture fast pattern matches and slower LLM-as-judge calls in the same histogram. INJECTION_DETECTIONS carries three labels—guard_type, vector, and severity—which power the Grafana attack-analysis row's time-series graph of detection rate by vector type and its severity distribution pie chart. ACTIVE_GUARDS is the health signal confirming all expected guards loaded after a configuration change.

With metrics in place, Prometheus alert rules convert raw numbers into actionable notifications. The rule below fires when the five-minute detection rate crosses 50 per minute for two consecutive minutes—the threshold above which a coordinated attack is the most likely explanation:

Code snippetyaml
1groups:
2  - name: injection_defense_alerts
3    rules:
4      - alert: HighInjectionRate
5        expr: rate(guard_injection_detections_total[5m]) > 50
6        for: 2m
7        labels:
8          severity: critical
9        annotations:
10          summary: "High injection attempt rate detected"

The for: 2m clause prevents flapping on brief spikes. The severity: critical label routes this alert to the on-call channel while lower-priority warning rules go to a slower-response queue.

Once alerts are live, the same metrics feed the defense-tuning cycle. If guard_evaluation_duration_seconds shows the LLM-as-judge consistently adding 300 ms but INJECTION_DETECTIONS shows it catches only 5 % of incremental attacks beyond what the pattern guard already blocked, raising the short-circuit threshold for the LLM judge is data-justified rather than guesswork—exactly the tuning loop the Concepts section described.

Confirm that Prometheus scrapes the /metrics endpoint without error and that at least one guard_injection_detections_total data point appears in the Grafana detection-rate panel after sending a test injection string through the guard chain.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do label INJECTION_DETECTIONS with all three dimensions (guard_type, vector, severity) — these three labels are what drive the Grafana attack-analysis row's detection-rate-by-vector time series and severity distribution pie chart; collapsing any one label makes it impossible to distinguish which guard caught which attack class at what urgency, losing the threat-intelligence signal entirely.
2. ✓Do set custom histogram buckets spanning 1 ms to 1 s for guard_evaluation_duration_seconds — the default Prometheus buckets are tuned for HTTP handler latency and will under-sample both ends of a guard chain that mixes sub-10 ms pattern matches with 100–500 ms LLM-as-judge calls; the custom [0.001 … 1.0] range keeps both guard types visible in the same histogram.
3. ✓Do cross-reference guard_evaluation_duration_seconds with guard_injection_detections_total before raising the LLM-as-judge short-circuit threshold — if the histogram shows the judge consistently adding 300 ms while its incremental detection gain over the pattern guard is only ~5 %, that pairing gives data-justified grounds to tighten the threshold rather than making the change as guesswork.

Don'ts

1. ✗Don't omit the result label from SCAN_REQUESTS — without splitting by result: "allowed" versus result: "blocked", you cannot compute a per-endpoint block rate, which is the primary signal in the operational-awareness dashboard row; all traffic looks identical and the gauge becomes uninterpretable during an active attack.
2. ✗Don't use a raw counter threshold in the HighInjectionRate alert expression — guard_injection_detections_total > 50 queries a monotonically increasing value that fires permanently once fifty injections have ever been seen; the correct form is rate(guard_injection_detections_total[5m]) > 50, which measures current velocity and triggers only when the rate is actively elevated.
3. ✗Don't drop the for: 2m clause from alert rules — removing it means any two-second spike in detection count that crosses the 50/min threshold triggers a critical page to the on-call team; the two-minute hold is the explicit mechanism that separates genuine coordinated attacks from transient traffic bursts, and its absence causes alert fatigue that trains engineers to ignore real escalations.