Manage K8s secrets with rotation and init-container injection

Introduction

Engineers often discover that AI deployments in customer Kubernetes clusters fail compliance reviews not because of faulty models, but because credentials are hardcoded, never rotated, or scattered across plaintext ConfigMaps. Customer environments demand automated, auditable credential lifecycle management—secrets must be namespace-scoped, annotated with rotation deadlines, and surfaced for operator review before audits expose drift. By the end of this lesson, you'll be able to create time-bound Kubernetes secrets with Base64-encoded data and TTL annotations, and understand how to track rotation schedules so your deployment automation can enforce security policies without manual credential management.

Key Terminology

• Kubernetes Secret — A namespace-scoped API object (V1Secret) that stores credential data as Base64-encoded key-value pairs, keeping sensitive values out of ConfigMaps and pod specs while making them injectable as environment variables or volume mounts.
• Base64 encoding — A binary-to-text serialization step required by the Kubernetes Secret API; the SecretsManager.create_secret method applies base64.b64encode(v.encode()).decode() to every plain-text value in spec.data before constructing the V1Secret body.
• TTL annotation — A metadata annotation (keyed rotation-due) stamped onto each Secret at creation time, holding an ISO-formatted UTC timestamp derived from datetime.utcnow() + timedelta(hours=spec.ttl_hours); it encodes the rotation deadline directly on the object so no external database is required.
• SecretSpec — A Pydantic BaseModel that validates and normalizes secret parameters—name, namespace, plain-text data, ttl_hours, optional labels, and secret_type—before the SecretsManager encodes or writes anything to the cluster.
• Label selector — The managed-by: ai-deployer label applied to every secret created by SecretsManager; it makes the full set of deployment-owned secrets in a namespace queryable with a single label-filtered list_namespaced_secret call, which is the hook a rotation-scanner job needs to sweep for overdue credentials.
• load_incluster_config() — A Kubernetes Python client call that reads the pod's mounted service-account token and cluster CA certificate to authenticate API calls from inside the cluster, replacing the kubeconfig lookup used in local development.

Concepts

Kubernetes Secrets are storage, not lifecycle management

A Kubernetes Secret solves exactly one problem: it keeps credential bytes out of ConfigMaps, pod specs, and image layers, and makes them injectable into workloads as environment variables or volume mounts. What it does not provide is any notion of expiry, rotation deadlines, or ownership. Left alone, a Secret created today is indistinguishable from one created two years ago—both sit in etcd with no signal that one has drifted past a compliance deadline.

Customer K8s environments that host AI deployments almost always carry external audit requirements: credentials must rotate on a defined schedule, and operators must be able to see—before an auditor does—which secrets are overdue. The SecretsManager class addresses this gap by treating the Secret object itself as the single source of truth for its own rotation schedule, using annotations to co-locate the deadline with the credential (see Code Walkthrough).

Base64 is a serialization contract, not a security boundary

A common misconception is that Base64-encoding a value inside a Kubernetes Secret "protects" it. It does not. Base64 is a reversible encoding that satisfies the Secret API's requirement to store arbitrary binary data as a string field—nothing more. Any principal with get permission on the Secret can decode the value in a single command.

This distinction matters for deployment automation: the create_secret method applies base64.b64encode(v.encode()).decode() to every plain-text credential in spec.data purely to satisfy the API contract, not to obscure the value. The real access control is Kubernetes RBAC scoped to the namespace—granting only the deployer service account create permission on Secrets in the customer namespace, and granting operators only list/get on secrets they need to inspect.

Annotations as a rotation-schedule ledger

Rather than maintaining a separate database of "which secret expires when," SecretsManager stamps the rotation deadline directly onto the Secret's metadata.annotations block as rotation-due. The timestamp is computed at write time: datetime.utcnow() + timedelta(hours=spec.ttl_hours), where ttl_hours defaults to 720 (30 days) but can be tightened to 168 or fewer for high-sensitivity namespaces.

This design keeps the lifecycle metadata co-located with the credential object, which means a rotation-scanner job—running as a scheduled CronJob inside the customer cluster—can discover every deployment-owned secret using the managed-by: ai-deployer label selector and evaluate expiry by reading the rotation-due annotation, all without querying any external system. The create_secret method returns the secret name and rotation_due ISO string so orchestration code can log or persist the schedule at grant time without a follow-up cluster query (see Code Walkthrough).

Code Walkthrough

Now that you understand how Kubernetes Secrets store Base64-encoded credentials and how TTL annotations drive rotation scheduling, the SecretsManager class below puts both mechanics into a single deployable unit.

The class pairs a SecretSpec validation model with a SecretsManager that wraps CoreV1Api. When create_secret is called, it Base64-encodes each plain-text credential value—a hard requirement of the Kubernetes Secret API—calculates a rotation_due timestamp from ttl_hours, and stamps that timestamp as an annotation on the created Secret. The managed-by: ai-deployer label makes every managed secret selectable by label, which is the hook a rotation-scanner job needs to list all deployment-owned secrets across a namespace without scanning unrelated objects.

Code snippetpython
1from kubernetes import client, config
2from pydantic import BaseModel, Field
3from datetime import datetime, timedelta
4from typing import Dict
5import base64
6
7class SecretSpec(BaseModel):
8    name: str
9    namespace: str
10    data: Dict[str, str]
11    ttl_hours: int = Field(default=720, description="Hours until rotation required")
12    labels: Dict[str, str] = {}
13    secret_type: str = "Opaque"
14
15class SecretsManager:
16    def __init__(self):
17        config.load_incluster_config()
18        self.core_v1 = client.CoreV1Api()
19
20    def create_secret(self, spec: SecretSpec) -> dict:
21        encoded_data = {
22            k: base64.b64encode(v.encode()).decode()
23            for k, v in spec.data.items()
24        }
25        rotation_due = datetime.utcnow() + timedelta(hours=spec.ttl_hours)
26        secret = client.V1Secret(
27            api_version="v1",
28            kind="Secret",
29            metadata=client.V1ObjectMeta(
30                name=spec.name,
31                namespace=spec.namespace,
32                labels={**spec.labels, "managed-by": "ai-deployer"},
33                annotations={
34                    "rotation-due": rotation_due.isoformat(),
35                    "created-by": "secrets-manager",
36                    "ttl-hours": str(spec.ttl_hours),
37                },
38            ),
39            type=spec.secret_type,
40            data=encoded_data,
41        )
42        self.core_v1.create_namespaced_secret(namespace=spec.namespace, body=secret)
43        return {"name": spec.name, "rotation_due": rotation_due.isoformat()}

The ttl_hours default of 720 (30 days) suits many LLM provider API keys, but customer security policies vary—high-sensitivity namespaces commonly require 168 hours (7 days) or less. Pass a tighter value in the SecretSpec to match the customer's policy without touching any other code path. The returned dictionary surfaces the secret name and ISO-formatted rotation deadline so orchestration code can log or persist the schedule without a follow-up cluster query.

To query which secrets are approaching their deadline, filter the namespace by the managed-by: ai-deployer label and read each secret's rotation-due annotation—the same field create_secret wrote. That pattern forms the basis of a rotation-status sweep you can run as a scheduled job inside the customer cluster, surfacing overdue credentials before they trigger a compliance finding.

Confirm that calling create_secret with a valid SecretSpec returns without error and that kubectl get secret <name> -n <namespace> -o yaml shows the rotation-due annotation and properly Base64-encoded values in the data block.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do Base64-encode every credential value via base64.b64encode(v.encode()).decode() — the Kubernetes Secret API requires Base64-encoded strings in the data block; submitting plaintext raises a validation error from the API server and your secret is never created.
2. ✓Do set ttl_hours in SecretSpec to match the customer's rotation policy — the rotation_due annotation stamped on creation is the only timestamp a rotation-scanner job has to work with, so a miscalibrated TTL (e.g., leaving the 720-hour default in a namespace that requires 168 hours) causes the scanner to surface no overdue secrets even as credentials age past the customer's compliance threshold.
3. ✓Do label every managed secret with managed-by: ai-deployer — SecretsManager applies this label so a rotation-scanner job can use a label selector to list all deployment-owned secrets in a namespace without touching unrelated objects; omitting it means your sweep query returns nothing and overdue credentials go undetected before an audit.

Don'ts

1. ✗Don't store plaintext credentials in a Kubernetes ConfigMap — ConfigMaps are unencrypted and not subject to Kubernetes RBAC Secret-access controls; customer compliance reviews flag any credential found in a ConfigMap as an immediate finding, and there is no rotation-due annotation path to attach a lifecycle to them.
2. ✗Don't call config.load_incluster_config() in environments where the pod lacks a mounted service-account token — outside a running pod (e.g., local dev or CI), load_incluster_config raises ConfigException; use config.load_kube_config() in those contexts or the SecretsManager will fail to initialize before it ever touches the cluster.
3. ✗Don't rely on the returned rotation_due value as the authoritative deadline without also reading the rotation-due annotation back from the cluster — if the create_namespaced_secret call fails after the timestamp is calculated but before the object is persisted, the returned dict holds a deadline for a secret that does not exist; verifying via kubectl get secret … -o yaml (as the walkthrough prescribes) confirms the annotation was actually written.