Configure CI to run on GKE self-hosted runners

Introduction

When your CI needs GPU access, in-cluster network reachability to private services, or predictable cost at scale, GitHub-hosted runners stop being viable for a GenAI platform. Workflows either fail because they can't reach VPC-internal endpoints, or they burn money waiting on cold GPU spin-ups that never materialize. Deploying actions-runner-controller (ARC) onto the same GKE cluster that runs the platform gives you runners that pull from internal Artifact Registry, reach private services directly, and burst onto pre-warmed GPU node pools — at the cost of owning runner uptime, autoscaling, and security hardening yourself.

By the end of this lesson you'll be able to configure ARC on GKE with an autoscaling RunnerScaleSet, target the right node pool for GPU jobs, and bind runner pods to a least-privilege GCP service account via Workload Identity so CI authenticates without static keys.

Key Terminology

• actions-runner-controller (ARC) — a Kubernetes operator that creates ephemeral runner pods on demand and registers them with GitHub; it is the control plane that makes self-hosted runners on GKE viable at all.
• AutoscalingRunnerSet — ARC's CRD defining a pool's image, resources, labels, and min/max scaling; in this lesson it is the unit you configure to keep idle cost at zero while still bursting to GPU capacity.
• Runner group — GitHub's org-level grouping that controls which repos can target a runner pool via labels; it is how you stop a low-trust repo from scheduling jobs on the GPU pool.
• GitHub App — the auth mechanism ARC uses to register runners; preferred over PATs because permissions scope per-repo and rotate cleanly, which matters when the runner has GCP IAM bound to it.
• Workload Identity — GKE's binding from a Kubernetes service account to a GCP IAM service account; in this lesson it is what lets runner pods get GCP credentials without baking a static JSON key into the image.

Concepts

ARC architecture and ephemeral pods

ARC turns each CI job into a fresh pod. A webhook triggers the controller, which creates a runner pod that registers with GitHub, receives a single job, runs it, and terminates. Pods are ephemeral — there is no persistent runner state between jobs, which closes the most common security weakness in self-hosted pools (long-lived runners accumulating artifacts from compromised PRs).

Autoscaling to zero with RunnerScaleSets

The pool is declared as an AutoscalingRunnerSet with minRunners: 0 and a maxRunners cap. With min at zero, a quiet weekend costs nothing; cold-start latency is ~30s, which is acceptable for almost all CI flows. The runner image is pinned monthly and pre-bakes heavy Python deps (PyTorch, transformers) so individual jobs don't re-install them. GPU pools add a nodeSelector on cloud.google.com/gke-accelerator and a toleration for the nvidia.com/gpu=NoSchedule taint so runner pods land on GPU nodes (see Code Walkthrough).

Workload Identity for keyless GCP auth

Runner pods need to pull from Artifact Registry, read provider keys from Secret Manager, and write to CI cache buckets. Instead of mounting a JSON key, the Kubernetes service account is bound to a GCP IAM service account via Workload Identity. The runner uses google.auth.default() to pick up an OIDC-derived token. No key to rotate, no key to leak from a compromised PR, and the GCP service account's IAM scope becomes the only security boundary that matters (see Code Walkthrough).

Runner group isolation

A single runner group like ai-platform-gpu collects pools that share an IAM and trust boundary. Workflows opt in via runs-on: gpu-runner. The group's repo allowlist prevents a low-trust repo from scheduling onto a GPU pool whose service account can read production secrets. CPU and GPU runners live in different groups with different IAM bindings — a flaky CPU test never monopolizes GPU capacity, and a CPU runner compromise never reaches GPU-tier secrets.

Code Walkthrough

The two snippets below demonstrate the autoscaling pool declaration and the Workload Identity binding that gives those pods keyless GCP access — the two artifacts that, together, realize every concept above.

Code snippetyaml
1apiVersion: actions.github.com/v1alpha1
2kind: AutoscalingRunnerSet
3metadata:
4  name: gke-gpu-runners
5  namespace: arc-runners
6spec:
7  githubConfigUrl: https://github.com/ai-platform
8  githubConfigSecret: gh-app-secret
9  minRunners: 0
10  maxRunners: 10
11  runnerGroup: ai-platform-gpu
12  template:
13    spec:
14      serviceAccountName: arc-runner-sa
15      nodeSelector:
16        cloud.google.com/gke-accelerator: nvidia-l4
17      tolerations:
18        - key: nvidia.com/gpu
19          operator: Exists
20      containers:
21        - name: runner
22          image: us-docker.pkg.dev/platform/ci/runner-gpu:2025.04
23          resources:
24            limits:
25              nvidia.com/gpu: 1
26              memory: 16Gi
27              cpu: 4

githubConfigSecret references a Kubernetes Secret with the GitHub App's private key and app id — that is what authenticates ARC to register runners. minRunners: 0 scales the pool to zero when idle; maxRunners: 10 caps the burst. runnerGroup ties the pool to the GitHub group whose repo allowlist gates which workflows can target it. The nodeSelector plus toleration land pods on GPU nodes; without the toleration the pod stays Pending. nvidia.com/gpu: 1 is the device-plugin request that exposes /dev/nvidia0 to the container — omit it and the pod gets CPU-only access even on a GPU node.

Code snippetbash
1gcloud iam service-accounts create ci-runner
2
3gcloud iam service-accounts add-iam-policy-binding \
4  ci-runner@platform.iam.gserviceaccount.com \
5  --role="roles/iam.workloadIdentityUser" \
6  --member="serviceAccount:platform.svc.id.goog[arc-runners/arc-runner-sa]"
7
8kubectl annotate serviceaccount arc-runner-sa \
9  --namespace arc-runners \
10  iam.gke.io/gcp-service-account=ci-runner@platform.iam.gserviceaccount.com
11
12gcloud projects add-iam-policy-binding platform \
13  --member="serviceAccount:ci-runner@platform.iam.gserviceaccount.com" \
14  --role="roles/artifactregistry.reader"
15
16gcloud projects add-iam-policy-binding platform \
17  --member="serviceAccount:ci-runner@platform.iam.gserviceaccount.com" \
18  --role="roles/secretmanager.secretAccessor"

The roles/iam.workloadIdentityUser binding lets the K8s service account arc-runner-sa in the arc-runners namespace impersonate the GCP service account. The annotation on the K8s service account completes the bind so google.auth.default() resolves to the GCP identity at runtime. The two project-level grants are the entire IAM footprint a CI runner should ever need — reader on Artifact Registry to pull images, secretAccessor on Secret Manager for provider keys.

You'll know it works when a workflow with runs-on: gpu-runner triggers an ARC pod that pulls the runner image from Artifact Registry, registers with the ai-platform-gpu group, runs the job, and terminates — verify with kubectl -n arc-runners get pods -w and confirm the pod disappears within seconds of the GitHub job completing.

Do's and Don'ts

Do's

1. ✓Do set minRunners: 0 — idle runners cost real money and the ~30s cold-start is acceptable for almost every CI flow.
2. ✓Do pin runner images by digest — a floating tag means tomorrow's CI runs different software than today's, with no audit trail.
3. ✓Do isolate GPU runners from CPU runners — separate node pools, runner groups, and IAM bindings so a flaky CPU test never monopolizes GPU capacity.

Don'ts

1. ✗Don't grant runners broad IAM roles — a single compromised PR runs arbitrary code in the runner; scope the GCP service account to only what tests legitimately need.
2. ✗Don't mount static GCP JSON keys — Workload Identity removes the entire class of leaked-key incidents and there is no operational reason to keep keys around.
3. ✗Don't share runner groups across trust boundaries — never run untrusted PR workflows on the same group as protected-branch workflows.