Deploy security monitoring stack on GKE

Introduction

When you run AI workloads on GKE without dedicated security observability, attacks, agent behavioral anomalies, and threat-intelligence matches stay invisible until a breach surfaces them. Setting up Prometheus, Grafana, and custom exporters by hand leaves configuration scattered across YAML files that drift from each other and from the actual application behavior. By the end of this lesson, you will be able to deploy the full security monitoring stack on GKE using Helm, with Python-generated Kubernetes manifests and Helm values that encode your detection rules, alert thresholds, and Grafana dashboard panels as version-controlled, programmatically validated configuration.

Key Terminology

• Counter — a monotonically increasing metric that records cumulative totals; Prometheus derives rates from it using rate() or irate(). In this lesson, SECURITY_EVENTS and DETECTIONS are Counters labeled by dimensions such as event_type, severity, and rule_id.
• Gauge — a metric that records a single instantaneous value that can rise or fall, suitable for quantities like per-agent behavioral drift scores. AGENT_DRIFT is a Gauge keyed by ["agent_id"] so the current drift reading for each monitored agent is independently tracked.
• Histogram — a metric that records observed values in pre-declared buckets, enabling percentile queries over the collected distribution. DETECTION_LATENCY uses a Histogram with buckets spanning 1 ms to 10 s to capture the full latency range of the detection pipeline across different detector_type values.
• Metric labels — key-value dimensions declared alongside a metric name (e.g., ["event_type", "severity", "source"] on SECURITY_EVENTS) that let Prometheus split and aggregate a single metric across multiple independent streams without requiring separate metric names for each combination.
• Scrape annotation — a Kubernetes pod annotation (prometheus.io/scrape: "true" paired with prometheus.io/port) that tells Prometheus to auto-discover a pod as a scrape target on the next sync cycle, removing the need for any manual job entry in Helm values.
• Python-generated manifest — the pattern of constructing a Kubernetes deployment as a Python dictionary inside generate_exporter_deployment and serializing it with yaml.dump, so that resource limits, scrape annotations, and container ports are always derived from the same parameterized function call and reviewable as deterministic diffs.

Concepts

Choosing the Right Prometheus Metric Type

Prometheus provides three distinct primitive types, each with a contract you must understand before writing metric definitions. A Counter only ever increases; Prometheus derives rates from its cumulative total, making it the correct type any time you want "events per second" — security events ingested or detection rules fired. A Gauge records the current reading of a quantity that fluctuates freely in both directions — agent behavioral drift scores rise and fall as behavior shifts, so AGENT_DRIFT is a Gauge rather than a Counter. A Histogram pre-declares value buckets and accumulates counts per bucket, enabling percentile queries (histogram_quantile) after the fact; DETECTION_LATENCY uses this form to answer "what is the p95 detection latency for rule-based detectors?"

Picking the wrong type produces silent errors rather than noisy failures: a Gauge for cumulative events loses rate semantics; a Counter for drift scores cannot decrease. The bucket boundaries on DETECTION_LATENCY — [0.001, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0] — are a deliberate design choice, not a default. Buckets must span the actual distribution: if all observations land inside two adjacent buckets the histogram provides no resolution. Tuning buckets for security workloads means covering both sub-millisecond in-process detections and multi-second anomaly correlation passes (see Code Walkthrough).

Python as the Configuration Source of Truth

Security monitoring stacks written as static YAML drift because the exporter deployment, the Prometheus scrape configuration, and the container port live in separate files with no enforced relationship. When an operator changes the exporter port in the deployment YAML but forgets the Prometheus job config, scraping breaks silently.

The lesson's design decision is that Python generates every Kubernetes manifest. generate_exporter_deployment is a plain function that accepts a namespace, image tag, and port, and returns a complete deployment dictionary. Calling it with different arguments — a staging namespace, a canary image tag — always produces identical structure with only the parameterized fields varying. That determinism makes diff-based review meaningful: the reviewer sees the exact YAML delta before it reaches the cluster, not an inferred "something probably changed." Because the port value flows into both containerPort and the prometheus.io/port annotation inside a single call, the two fields can never disagree (see Code Walkthrough).

Automatic Scrape Target Discovery via Pod Annotations

Prometheus discovers scrape targets through annotations on Kubernetes pods rather than through static job definitions in its configuration. The annotations prometheus.io/scrape: "true" and prometheus.io/port: "<port>" instruct Prometheus to poll a pod's /metrics endpoint on the next sync cycle, with no corresponding entry required in the Helm values file.

For the security exporter this means there is no separate configuration block that must be kept synchronized with the deployment spec. When generate_exporter_deployment writes the port into the pod annotation, Prometheus picks up the target automatically on its next Kubernetes service-discovery sync. The practical verification is simple: after running kubectl apply -f exporter-deployment.yaml, check that security-metrics-exporter appears as an UP target in the Prometheus targets UI. A target listed as UP confirms both that the pod is reachable and that the annotation was read correctly — failure to appear means either the pod is not yet Running or the annotation value does not match the port the exporter is listening on.

Code Walkthrough

Now that you understand the three-component architecture — Prometheus, Grafana, and the custom Python exporters — and the principle that Python generates all Kubernetes configuration, the following walkthrough shows those two ideas working together in the exporter layer.

The exporter defines its metrics using prometheus_client and separately produces its own GKE deployment manifest as a Python dictionary that you serialize to YAML before applying to the cluster:

Code snippetpython
1from prometheus_client import Counter, Gauge, Histogram, start_http_server
2import yaml
3
4# Prometheus metric definitions for the security stack
5SECURITY_EVENTS = Counter(
6    "ai_security_events_total",
7    "Total security events ingested",
8    ["event_type", "severity", "source"],
9)
10DETECTIONS = Counter(
11    "ai_security_detections_total",
12    "Total rule-engine detections fired",
13    ["rule_id", "severity", "mitre_atlas_id"],
14)
15AGENT_DRIFT = Gauge(
16    "ai_agent_drift_score",
17    "Current behavioral drift score for a monitored agent",
18    ["agent_id"],
19)
20DETECTION_LATENCY = Histogram(
21    "ai_security_detection_latency_seconds",
22    "Detection pipeline latency in seconds",
23    ["detector_type"],
24    buckets=[0.001, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0],
25)

Counters accumulate totals that Prometheus turns into rates; Gauges track instantaneous values like per-agent drift scores; the Histogram captures detection-latency distributions across bucket boundaries tuned for security workloads (1 ms to 10 s).

The exporter's GKE deployment is not written as a static YAML file. A Python function generates it, keeping resource limits, scrape annotations, and container ports in sync with the metric definitions:

Code snippetpython
1def generate_exporter_deployment(namespace: str, image: str, port: int) -> dict:
2    return {
3        "apiVersion": "apps/v1",
4        "kind": "Deployment",
5        "metadata": {"name": "security-metrics-exporter", "namespace": namespace},
6        "spec": {
7            "replicas": 1,
8            "selector": {"matchLabels": {"app": "security-metrics-exporter"}},
9            "template": {
10                "metadata": {
11                    "labels": {"app": "security-metrics-exporter"},
12                    "annotations": {
13                        "prometheus.io/scrape": "true",
14                        "prometheus.io/port": str(port),
15                    },
16                },
17                "spec": {
18                    "containers": [{
19                        "name": "exporter",
20                        "image": image,
21                        "ports": [{"containerPort": port}],
22                        "resources": {
23                            "requests": {"cpu": "100m", "memory": "128Mi"},
24                            "limits":   {"cpu": "500m", "memory": "512Mi"},
25                        },
26                    }],
27                },
28            },
29        },
30    }
31
32if __name__ == "__main__":
33    manifest = generate_exporter_deployment(
34        namespace="security-monitoring",
35        image="gcr.io/my-project/security-exporter:v1",
36        port=9090,
37    )
38    with open("exporter-deployment.yaml", "w") as f:
39        yaml.dump(manifest, f, default_flow_style=False)
40    start_http_server(9090)

Calling generate_exporter_deployment with different parameters — staging versus production namespaces, different image tags — always produces deterministic output, so diff-based review catches every configuration change before it reaches the cluster. The prometheus.io/scrape and prometheus.io/port annotations are what Prometheus reads to discover the exporter as a scrape target automatically, without any manual job configuration in the Helm values.

Confirm that after running kubectl apply -f exporter-deployment.yaml, the pod reaches Running state (kubectl get pods -n security-monitoring), the /metrics endpoint returns Prometheus-formatted text, and Prometheus shows the security-metrics-exporter target as UP in its targets UI.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do generate the exporter's GKE deployment manifest from a Python function — generate_exporter_deployment(namespace, image, port) keeps resource limits, scrape annotations, and containerPort co-located and parameterized, so staging-vs-production differences are expressed as arguments rather than copy-pasted YAML files that drift from each other.
2. ✓Do choose metric types based on what Prometheus does with the data — use Counter for ai_security_events_total and ai_security_detections_total (Prometheus computes rates from accumulated totals), Gauge for ai_agent_drift_score (instantaneous per-agent values), and Histogram with domain-tuned buckets=[0.001, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0] for ai_security_detection_latency_seconds so every bucket boundary maps to a meaningful threshold in your detection pipeline.
3. ✓Do include prometheus.io/scrape: "true" and prometheus.io/port in the pod template annotations inside generate_exporter_deployment — these annotations are what Prometheus reads to auto-discover the security-metrics-exporter as a scrape target; omitting them means Prometheus never adds the exporter to its targets list and all metric definitions go uncollected.

Don'ts

1. ✗Don't write the exporter deployment as a static YAML file — a hand-authored exporter-deployment.yaml lets scrape annotations, image tags, and resource limits drift independently; the Python-generated manifest is the only shape where changing the port argument propagates to containerPort and prometheus.io/port simultaneously in one diff.
2. ✗Don't use the default prometheus_client Histogram buckets for security detection latency — the defaults are calibrated for HTTP request durations and will coarsely bin the 1 ms–100 ms range where most rule-engine detections land; always pass buckets=[0.001, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0] to DETECTION_LATENCY to preserve resolution across the full 1 ms–10 s operational range.
3. ✗Don't skip the three-step post-apply verification — after kubectl apply -f exporter-deployment.yaml, confirm pod Running state with kubectl get pods -n security-monitoring, hit the /metrics endpoint for Prometheus-formatted text, and check that security-metrics-exporter appears as UP in the Prometheus targets UI; a pod that starts but lacks the prometheus.io/scrape annotation fails silently with no metric data and no visible error.