Monitor GKE security posture continuously

Introduction

In production, a GKE cluster that passes every security review at launch can silently drift — a namespace added without PSA labels, a permissive NetworkPolicy left over from a debugging session, a Falco rule silenced to reduce noise. Each gap is individually small; collectively they erode the controls you put in place when hardening the cluster. By the end of this lesson, you'll be able to run an hourly posture collector that detects PSA, NetworkPolicy, Workload Identity, image-signing, and Falco-rule drift across every domain, emit per-domain scores to Grafana, and trigger an on-call alert the moment any domain score drops below 95 %.

Key Terminology

• Security Posture Drift — The gradual, silent erosion of cluster security controls after an initial hardening pass; examples include namespaces added without the pod-security.kubernetes.io/enforce: restricted label, allow-all NetworkPolicy rules left over from debugging sessions, and Falco rules silenced to reduce noise.
• Finding — A structured record emitted by PostureCollector that represents one detected control violation, carrying a domain (e.g., "psa" or "netpol"), a severity ("HIGH", "MEDIUM", or "LOW"), the affected target resource or namespace, and a human-readable detail string.
• Domain Score — A per-control-area compliance percentage computed as the ratio of compliant resources to total resources in scope (e.g., (enforced_namespaces / total_namespaces) × 100 for the PSA domain); each domain emits its score as a Prometheus metric scraped every 60 seconds and rendered as a 30-day trend line in Grafana.
• PostureCollector — The Python class that runs as a Kubernetes CronJob, invoking _namespace_psa_audit() and _network_policy_audit() on every cycle, merging results into a flat list of Finding objects, and exposing them as Prometheus metrics on a /metrics endpoint.
• EXCLUDED_NS — The set of Kubernetes system namespaces (kube-system, kube-public, kube-node-lease) that _namespace_psa_audit() skips when computing the PSA domain score, because they are control-plane-managed and not subject to user-defined enforcement labels.
• Allow-All Ingress Pattern — A NetworkPolicy ingress rule that specifies neither a from selector nor any ports, permitting unrestricted inbound traffic; _network_policy_audit() flags these as HIGH-severity netpol findings because they are the most common artifact left behind after a debugging session.

Concepts

Why Static Audits Leave Gaps

A GKE cluster that passes every security review at launch is secure at that instant — not permanently. Namespaces accumulate without PSA labels when teams work under deadline pressure. An ingress rule is widened to allow-all during an incident and never narrowed back. A Falco rule is silenced to quiet a noisy alert and forgotten. Each change is individually defensible; collectively they hollow out the controls that took weeks to establish.

Static audits — a quarterly review, a one-off kubectl scan — only snapshot posture at a single moment. The window between audits is exactly where drift compounds unseen. Continuous posture monitoring closes that window by re-measuring compliance every hour, surfacing regressions within a single SRE shift rather than a quarter.

The Collector-to-Grafana Pipeline

PostureCollector runs as a Kubernetes CronJob. On each execution its collect() method calls a suite of domain-specific audit functions — _namespace_psa_audit() for PSA enforcement, _network_policy_audit() for overly permissive rules — and merges the results into a flat list of Finding objects. Each finding carries a domain, severity, and the specific target that failed (see Code Walkthrough).

The collector exposes a /metrics endpoint in Prometheus exposition format. A scrape job polls it every 60 seconds; the data flows into Grafana, which renders a 30-day trend line per domain. No human needs to remember to run an audit — the system audits itself continuously, and the accumulated data makes gradual erosion visible at a glance.

Per-Domain Scores and Alert Thresholds

Each domain translates compliance into a single percentage: the ratio of resources in the desired state to the total count of in-scope resources. The PSA score is (enforced_namespaces / total_namespaces) × 100, where "enforced" means the pod-security.kubernetes.io/enforce label is present and set to restricted. Image-signing, Workload Identity, and Falco-rule domains follow the same pattern with their own numerators and denominators.

Two alert thresholds govern the board. A week-over-week drop of more than 5 % on any domain fires a warning — it signals gradual erosion even when the absolute score is still high. A domain score falling below 95 % pages on-call immediately, because at that level a meaningful fraction of workloads are unprotected. Treating the aggregate score as the only signal is a common pitfall: a single HIGH-severity finding can hide inside a 97 % score. The Finding list is the ground truth; the trend line is the early-warning system. Always inspect both.

Exception Hygiene and Namespace Exclusions

EXCLUDED_NS tells _namespace_psa_audit() to skip control-plane namespaces that the cluster manages internally; including them would inflate the denominator with resources that users cannot configure. For every other namespace, a missing or non-restricted PSA enforce label generates a HIGH finding immediately.

The same discipline applies to any exceptions carved out elsewhere in the collector. Every exception must carry an expiration date and a named approver. Exceptions without these fields accumulate over time and become structurally identical to the debugging-session artifacts the collector was built to catch — allowed gaps that no one can justify and no one wants to remove.

Code Walkthrough

Now that you've seen the per-domain score formulas and the operating discipline that governs them, this walkthrough shows the Python implementation that computes those scores against a live GKE cluster every hour.

The PostureCollector runs as a Kubernetes CronJob. Its collect() method calls a suite of audits, merges the results into a flat list of Finding objects, and emits per-domain Prometheus metrics that Grafana renders into the 30-day trend lines described in the Concepts section.

Code snippetpython
1from dataclasses import dataclass
2from kubernetes import client, config as k8s_config
3
4EXCLUDED_NS = {"kube-system", "kube-public", "kube-node-lease"}
5
6@dataclass
7class Finding:
8    domain: str      # "psa" | "netpol" | "image_signing" | "workload_id" | "falco"
9    severity: str    # "HIGH" | "MEDIUM" | "LOW"
10    target: str      # namespace or resource name
11    detail: str
12
13class PostureCollector:
14    def __init__(self):
15        k8s_config.load_incluster_config()
16        self.core = client.CoreV1Api()
17        self.net  = client.NetworkingV1Api()
18
19    def collect(self) -> list[Finding]:
20        findings: list[Finding] = []
21        findings.extend(self._namespace_psa_audit())
22        findings.extend(self._network_policy_audit())
23        return findings
24
25    def _namespace_psa_audit(self) -> list[Finding]:
26        results = []
27        for ns in self.core.list_namespace().items:
28            name = ns.metadata.name
29            if name in EXCLUDED_NS:
30                continue
31            psa = (ns.metadata.labels or {}).get(
32                "pod-security.kubernetes.io/enforce"
33            )
34            if psa != "restricted":
35                results.append(Finding(
36                    domain="psa", severity="HIGH", target=name,
37                    detail=f"PSA enforce={psa!r}, expected 'restricted'",
38                ))
39        return results
40
41    def _network_policy_audit(self) -> list[Finding]:
42        results = []
43        for np in self.net.list_network_policy_for_all_namespaces().items:
44            for rule in (np.spec.ingress or []):
45                if not rule._from and not rule.ports:
46                    results.append(Finding(
47                        domain="netpol", severity="HIGH",
48                        target=f"{np.metadata.namespace}/{np.metadata.name}",
49                        detail="ingress allows all — no from-selector and no ports",
50                    ))
51        return results

_namespace_psa_audit drives the PSA domain score: every non-restricted namespace outside EXCLUDED_NS becomes a HIGH finding, directly feeding the (enforced_namespaces / total_namespaces) × 100 formula from the Concepts section. _network_policy_audit flags the canonical "allow-from-anywhere" pattern — an ingress rule with neither a from selector nor a port restriction — the exact drift mode the Concepts section identifies as the most common debugging-session artefact.

The collector exposes a /metrics endpoint; a Prometheus scrape job pulls it every 60 seconds and feeds the Grafana board that displays per-domain trend lines. A week-over-week drop of more than 5 % triggers a warning alert; any domain score falling below 95 % pages on-call.

Confirm that instantiating PostureCollector and calling collect() against a namespace missing the pod-security.kubernetes.io/enforce: restricted label returns at least one Finding with domain="psa" and severity="HIGH", and that a NetworkPolicy whose ingress rule has an empty _from list and no ports produces a matching Finding with domain="netpol".

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do exclude system namespaces via EXCLUDED_NS — kube-system, kube-public, and kube-node-lease are managed by GKE itself and will never carry pod-security.kubernetes.io/enforce: restricted; including them inflates your finding count and drives the PSA domain score below 95 % on every healthy cluster.
2. ✓Do model each gap as a typed Finding with domain, severity, and target fields — this flat structure lets collect() merge PSA, NetworkPolicy, Workload Identity, image-signing, and Falco results into one list that a single Prometheus emit loop converts to per-domain metrics without branching on audit type.
3. ✓Do treat an ingress rule with an empty _from list AND no ports as a HIGH finding — this exact combination is the "allow-from-anywhere" pattern left behind by debugging sessions; flagging only one condition misses the full open-ingress form that _network_policy_audit is specifically designed to catch.

Don'ts

1. ✗Don't hard-code the PSA enforce label check against any value other than "restricted" — accepting "baseline" or a None result silently passes namespaces that permit privileged containers, and the (enforced_namespaces / total_namespaces) × 100 score formula will read as green while your actual workload isolation is broken.
2. ✗Don't suppress or silence Falco-rule findings to reduce alert noise — the lesson identifies silenced Falco rules as one of the three primary drift modes; trading fewer pages for reduced detection coverage is exactly the erosion pattern the hourly PostureCollector CronJob is designed to surface before it becomes a security gap.
3. ✗Don't scrape the /metrics endpoint at intervals longer than 60 seconds — the alerting threshold is a domain score drop below 95 %, and a week-over-week warning fires on a 5 % decline; infrequent scrapes mean Grafana trend lines miss short-lived regressions that recover before the next pull, hiding real drift from the 30-day history.