Deploy incident response automation on GKE

Introduction

When you automate incident response workflows in Kubernetes, every containment action your scripts perform — rotating credentials, scaling down compromised pods, applying network policies — runs under a service account whose permissions are declared in your Helm chart's values.yaml. Without correctly scoped RBAC rules in that file, automation either fails silently because it lacks the verbs it needs, or creates a privilege-escalation risk because it was granted too much. By the end of this lesson, you'll be able to structure an RBAC block in values.yaml that gives incident-response automation exactly the Kubernetes access containment requires — no broader.

Key Terminology

• RBAC rule — a single apiGroups/resources/verbs policy entry in a Kubernetes Role that grants an identity permission to perform specific operations on specific object types; the rules: list in values.yaml is composed of one or more of these entries, each scoped to a distinct capability boundary.
• Service account — the Kubernetes identity under which automation pods run; the incident-responder service account declared via serviceAccount.name is what the cluster's RBAC engine checks when the incident-response service attempts any API call such as deleting a pod or creating a network policy.
• Namespace-scoped Role — a Role (as opposed to a ClusterRole) whose permissions apply only within a single namespace; setting clusterRole: false in values.yaml directs Helm to render a Role rather than a ClusterRole, preventing the service account from reaching workloads outside the incident-response namespace.
• Verb enumeration — the explicit list of allowed API operations (e.g., get, list, patch, delete) declared per resource type in an RBAC rule; specifying only the verbs a containment action genuinely requires is the mechanical expression of least-privilege in values.yaml.
• Vault delegation — the pattern of granting the responder service account only get on secrets while routing write and delete operations through HashiCorp Vault, so every credential mutation carries a separate audit trail and the Kubernetes RBAC surface for secrets remains read-only.
• read_namespaced_role — the Kubernetes Python client method on RbacAuthorizationV1Api that fetches the live Role object from the cluster, used immediately after helm upgrade to confirm the rendered manifest matches the rules: block declared in values.yaml before any containment workflow is trusted to run.

Concepts

RBAC Scoping as the Safety Contract for Automation

Incident-response automation is unusually powerful: it scales down pods, applies network policies, reads credentials, and deletes workloads — often unsupervised and under time pressure. Every one of those actions runs under a Kubernetes service account, and the service account's permissions are exactly what the RBAC engine enforces at runtime. If the account has too few verbs, the automation fails silently mid-containment and may leave a compromised workload running. If it has too many, a bug in the automation code — or an attacker who pivots into the automation pod — gains a privilege-escalation path across the cluster. The values.yaml RBAC block is where you resolve this tension deliberately, before the automation ever executes.

Translating Containment Actions into Policy Rules

The design process is a direct mapping: every containment action the automation must perform becomes one or more apiGroups/resources/verbs triples. Scaling down a compromised deployment requires patch and update on apps/deployments. Isolating a pod via network segmentation requires create and patch on networking.k8s.io/networkpolicies. Reading pod logs for forensic capture requires get on pods/log. Each action has a minimum verb set; the rules: list in values.yaml is the union of those minimum sets across all containment actions the service must execute (see Code Walkthrough).

The secrets boundary is a deliberate exception to this union logic: the responder service account receives only get, even though revocation might seem to require write access. Write and delete operations are instead delegated to Vault, where they produce a separate, tamper-evident audit trail. This split means the automation can validate a credential without being able to overwrite or wipe it — limiting the damage if the automation itself is compromised during a response.

Namespace Confinement and Blast Radius

A ClusterRole grants a given verb set across every namespace in the cluster; a namespace-scoped Role grants it only within the namespace where it lives. For incident-response automation this distinction is a hard safety boundary: a responder that can delete pods in any namespace can accidentally — or through attacker misuse — disrupt workloads that have nothing to do with the incident. Setting clusterRole: false in values.yaml directs Helm to render a Role, confining the incident-responder service account to the incident-response namespace and bounding the blast radius of both automation defects and lateral movement attempts.

Verifying the Live Manifest After Deployment

Helm values files and chart templates can drift silently: a Kustomize overlay, a default in _helpers.tpl, or a stale kubectl apply can override the Role your values.yaml declared without any obvious error. The read_namespaced_role call in the Python verification script fetches the live Role object from the cluster's RBAC authorization API immediately after helm upgrade, producing the actual apiGroups, resources, and verbs the cluster is enforcing (see Code Walkthrough). Comparing that output against your values.yaml declarations confirms the rendered policy matches your intent — any discrepancy, whether a missing verb or an unexpected resource, must be resolved before the containment automation is trusted.

Code Walkthrough

Now that you understand the containment actions an incident-response automation service must execute, the values.yaml RBAC block is where you translate those actions into Kubernetes policy — one apiGroups/resources/verbs rule per capability boundary.

A minimal values.yaml block for an incident-response service scoped to a single namespace looks like this:

Code snippetyaml
1# values.yaml — incident-response-automation Helm chart
2rbac:
3  create: true
4  serviceAccount:
5    name: incident-responder
6    annotations: {}
7  clusterRole: false          # namespace-scoped; prefer Role over ClusterRole
8  rules:
9    - apiGroups: [""]
10      resources: ["pods", "pods/log", "configmaps"]
11      verbs: ["get", "list", "watch", "delete"]
12    - apiGroups: ["apps"]
13      resources: ["deployments", "replicasets"]
14      verbs: ["get", "list", "patch", "update"]
15    - apiGroups: ["networking.k8s.io"]
16      resources: ["networkpolicies"]
17      verbs: ["get", "list", "create", "update", "patch"]
18    - apiGroups: [""]
19      resources: ["secrets"]
20      verbs: ["get"]           # read-only; revocation is delegated to Vault

Each rule entry follows the same three-field pattern: apiGroups selects the Kubernetes API surface, resources names the object types, and verbs enumerates the allowed operations. Keeping secrets under get-only means the responder service account can read credentials for validation but cannot write or delete them — write operations go through Vault, where every action is separately audited. Setting clusterRole: false confines all permissions to the incident-response namespace and prevents the service account from reaching workloads in other namespaces.

After running helm upgrade, verify that the chart rendered the Role as intended by inspecting the live manifest via the Kubernetes Python client:

Code snippetpython
1from kubernetes import client, config
2
3def list_role_rules(namespace: str, role_name: str) -> list[dict]:
4    config.load_incluster_config()
5    rbac_v1 = client.RbacAuthorizationV1Api()
6    role = rbac_v1.read_namespaced_role(name=role_name, namespace=namespace)
7    return [
8        {
9            "apiGroups": rule.api_groups,
10            "resources": rule.resources,
11            "verbs": rule.verbs,
12        }
13        for rule in (role.rules or [])
14    ]
15
16if __name__ == "__main__":
17    rules = list_role_rules(
18        namespace="incident-response",
19        role_name="incident-responder",
20    )
21    for rule in rules:
22        print(rule)

read_namespaced_role fetches the live Role object from the cluster's RBAC authorization API and returns each policy rule as a plain dictionary. Running this immediately after a Helm deployment confirms the rendered manifest matches your values.yaml intent before any containment workflow is triggered.

Confirm that the output of list_role_rules returns exactly the apiGroups, resources, and verbs entries declared in values.yaml — any discrepancy indicates a Helm template or Kustomize overlay has overridden your RBAC definition and must be resolved before the automation is trusted to run containment actions.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do set clusterRole: false in your values.yaml RBAC block — a namespace-scoped Role confines the incident-responder service account to the incident-response namespace, preventing it from reaching deployments, network policies, or secrets in unrelated namespaces; a ClusterRole widens the blast radius to the entire cluster the moment credentials are compromised.
2. ✓Do restrict the secrets rule to the get verb only and delegate write operations to Vault — allowing the responder service account only to read credentials for validation while routing revocation through Vault preserves a separate audit trail for every destructive credential action, which is essential for post-incident forensics.
3. ✓Do call read_namespaced_role via RbacAuthorizationV1Api immediately after every helm upgrade — comparing the live apiGroups, resources, and verbs entries against your values.yaml catches silent overrides from Helm templates or Kustomize overlays before any containment workflow is allowed to execute under a misconfigured permission set.

Don'ts

1. ✗Don't set clusterRole: true to simplify deployment — a ClusterRole lets the incident-responder service account patch deployments and create network policies across every namespace, turning a breach of that account into a cluster-wide privilege-escalation path rather than a contained namespace incident.
2. ✗Don't add create, update, or delete verbs to the secrets rule in values.yaml — granting the responder service account write access to secrets collapses the audit boundary Vault enforces; if the automation is compromised, an attacker can rotate or destroy credentials without leaving a Vault audit log entry.
3. ✗Don't assume the rendered Role on-cluster matches your values.yaml after helm upgrade — a discrepancy between what list_role_rules returns and what you declared is evidence that a Helm template or Kustomize overlay silently overrode your RBAC definition; running containment workflows before resolving that discrepancy means your isolation boundaries are untested.