Implement K8s namespace provisioning with quota enforcement

Introduction

When you onboard a new tenant to a multi-tenant Kubernetes platform, you need more than just a namespace — you need deterministic naming, resource quotas that match the tenant's tier, and limit ranges that prevent any single container from starving its neighbors. Engineers often wire this up ad-hoc, scattering quota values across scripts and YAML files until a tier change requires hunting down every reference. By the end of this lesson, you'll be able to implement a TenantProvisioner class that creates a namespace, applies tier-appropriate ResourceQuota and LimitRange objects, and handles 409 conflicts so the entire flow is safe to retry.

Key Terminology

• ResourceQuota — A Kubernetes API object that enforces aggregate hard limits on CPU, memory, GPU, and pod counts within a namespace; in TenantProvisioner, values from QuotaSpec fields like cpu_limit and pod_count are written into V1ResourceQuotaSpec.hard keys such as "limits.cpu" and "pods".
• LimitRange — A Kubernetes API object that injects per-container default resource requests and limits, preventing containers that omit resource declarations from claiming unbounded cluster capacity; built from LimitRangeSpec fields like default_cpu and default_request_memory and applied as a V1LimitRangeItem of type "Container".
• QuotaSpec / LimitRangeSpec — Frozen Python dataclasses that encode all resource values for a single tier; marking them frozen=True causes Python to raise a FrozenInstanceError on any attempted in-place mutation, making unauthorized quota changes a hard runtime failure rather than a silent data error.
• Tier registry — The TIER_QUOTAS dictionary that maps tier strings ("free", "standard", "enterprise") to (QuotaSpec, LimitRangeSpec) tuples; the quota_for_tier factory reads from it after normalizing the input and raises ValueError immediately — before any Kubernetes API call — on an unrecognized tier string.
• Idempotent provisioning — The property that calling TenantProvisioner.provision multiple times with the same arguments converges to the same cluster state; achieved by catching ApiException with status == 409 during namespace creation and treating it as a no-op, so retries after transient failures never leave the cluster in a broken partial state.
• Replace-on-conflict — The apply strategy used for ResourceQuota and LimitRange in provision: the method attempts replace_namespaced_resource_quota (and the limit-range equivalent) first and falls back to create_* only on a 404, so that re-invoking provision with a new tier string updates quotas in-place without a separate "update tier" code path.

Concepts

Three Kubernetes objects make one isolation boundary

A namespace alone does not enforce resource isolation — it is only a naming and access-control scope. Two additional objects, applied inside the namespace, supply the actual compute guardrails. A ResourceQuota sets a hard aggregate ceiling: if the pods running in a namespace would collectively exceed the quota's limits.cpu or pods count, the API server rejects the new pod at admission time, before it ever schedules. A LimitRange works at the per-container level: it injects default resource requests and limits into any container spec that omits them, closing the gap that would otherwise let a single container claim unbounded memory simply by declaring nothing.

These three objects must be created in strict order — namespace first, then ResourceQuota, then LimitRange — because the API server rejects quota or limit-range objects that reference a namespace that does not yet exist. That sequencing is not a convention; it is enforced by the server (see Code Walkthrough).

Idempotency through two distinct conflict patterns

Production provisioners are invoked from task queues, Kubernetes Jobs, or webhook handlers — all contexts where a transient network failure can trigger a retry mid-flight. If individual API calls are not safe to re-issue, a partial retry can leave the cluster with a namespace but no quota, or a quota that never got updated after a tier change.

The provisioner handles this with two different conflict responses. Namespace creation catches ApiException with status == 409 and treats it as success — a 409 means the namespace already exists, which is exactly the goal. For ResourceQuota and LimitRange, the method first attempts a replace_namespaced_* call, which always writes the caller-supplied values whether or not the object existed before. If the object is genuinely absent (a 404 response), the code falls back to create_namespaced_*. The replace-first strategy has a valuable side effect: re-invoking provision with a new tier string atomically updates the running quota without a separate "update" code path — the same method handles initial setup and tier upgrades (see Code Walkthrough).

Centralizing tier values in frozen dataclasses

Quota values that live in application code rather than scattered YAML files are easier to version-control, unit-test, and audit. The lesson centralizes all tier-specific numbers in the TIER_QUOTAS dictionary, where each key is a normalized tier string and each value is a (QuotaSpec, LimitRangeSpec) tuple. Both dataclasses are declared frozen=True, making them immutable value objects: any call site that attempts spec.cpu_limit = "128" gets a FrozenInstanceError immediately, catching unauthorized mutations at the earliest possible moment rather than silently propagating them into an API call.

The quota_for_tier factory performs normalization — strip and lowercase — before the dictionary lookup, and raises a descriptive ValueError for any unrecognized tier string. The error surfaces before any Kubernetes API call is attempted, producing a clear Python exception rather than an opaque API response that would arrive several round trips later.

Code Walkthrough

Now that you understand the key design decisions — deterministic tenant-{id} naming, frozen QuotaSpec/LimitRangeSpec dataclasses, and replace-on-conflict semantics — let's walk through a complete provisioner implementation.

The first block defines the data layer: frozen dataclasses for quota and limit-range values, a TIER_QUOTAS registry, and the quota_for_tier factory that raises ValueError on an unrecognized tier string.

Code snippetpython
1from dataclasses import dataclass
2from typing import Tuple
3
4@dataclass(frozen=True)
5class QuotaSpec:
6    cpu_limit: str
7    memory_limit: str
8    gpu_limit: str
9    pod_count: int
10
11@dataclass(frozen=True)
12class LimitRangeSpec:
13    default_cpu: str
14    default_memory: str
15    default_request_cpu: str
16    default_request_memory: str
17
18TIER_QUOTAS = {
19    "free": (
20        QuotaSpec(cpu_limit="4", memory_limit="8Gi", gpu_limit="0", pod_count=20),
21        LimitRangeSpec("200m", "256Mi", "100m", "128Mi"),
22    ),
23    "standard": (
24        QuotaSpec(cpu_limit="16", memory_limit="32Gi", gpu_limit="1", pod_count=100),
25        LimitRangeSpec("500m", "512Mi", "250m", "256Mi"),
26    ),
27    "enterprise": (
28        QuotaSpec(cpu_limit="64", memory_limit="128Gi", gpu_limit="4", pod_count=500),
29        LimitRangeSpec("1", "1Gi", "500m", "512Mi"),
30    ),
31}
32
33def quota_for_tier(tier: str) -> Tuple[QuotaSpec, LimitRangeSpec]:
34    tier_lower = tier.strip().lower()
35    if tier_lower not in TIER_QUOTAS:
36        raise ValueError(
37            f"Unknown tier '{tier}'. Must be one of: {list(TIER_QUOTAS)}"
38        )
39    return TIER_QUOTAS[tier_lower]

The second block shows the TenantProvisioner class. It receives a configured kubernetes.client.CoreV1Api instance and exposes a single provision method. The method derives the namespace name from the tenant ID, calls quota_for_tier to get the right specs, and then issues three sequential Kubernetes API calls — namespace creation first, because the API server rejects ResourceQuota and LimitRange objects that reference a non-existent namespace. Any ApiException with status 409 is caught and treated as success, making every step idempotent and safe to retry after a transient failure. When a tenant changes tiers, re-invoking provision with the new tier string replaces the quota and limit range without requiring a separate update path.

Code snippetpython
1from dataclasses import dataclass
2from typing import Tuple
3from kubernetes import client
4from kubernetes.client.rest import ApiException
5
6def quota_for_tier(tier: str) -> Tuple["QuotaSpec", "LimitRangeSpec"]:
7    # (defined above; repeated import context only)
8    ...
9
10class TenantProvisioner:
11    def __init__(self, core_v1: client.CoreV1Api) -> None:
12        self._api = core_v1
13
14    def provision(self, tenant_id: str, tier: str) -> dict:
15        namespace = f"tenant-{tenant_id}"
16        quota_spec, lr_spec = quota_for_tier(tier)
17
18        # 1. Create namespace
19        ns_body = client.V1Namespace(
20            metadata=client.V1ObjectMeta(
21                name=namespace,
22                labels={
23                    "platform/tenant-id": tenant_id,
24                    "platform/tier": tier.lower(),
25                },
26            )
27        )
28        try:
29            self._api.create_namespace(ns_body)
30        except ApiException as exc:
31            if exc.status != 409:
32                raise
33
34        # 2. Apply ResourceQuota
35        quota_body = client.V1ResourceQuota(
36            metadata=client.V1ObjectMeta(name="tenant-quota", namespace=namespace),
37            spec=client.V1ResourceQuotaSpec(
38                hard={
39                    "limits.cpu": quota_spec.cpu_limit,
40                    "limits.memory": quota_spec.memory_limit,
41                    "requests.nvidia.com/gpu": quota_spec.gpu_limit,
42                    "pods": str(quota_spec.pod_count),
43                }
44            ),
45        )
46        try:
47            self._api.replace_namespaced_resource_quota("tenant-quota", namespace, quota_body)
48        except ApiException as exc:
49            if exc.status == 404:
50                self._api.create_namespaced_resource_quota(namespace, quota_body)
51            else:
52                raise
53
54        # 3. Apply LimitRange
55        lr_body = client.V1LimitRange(
56            metadata=client.V1ObjectMeta(name="tenant-limits", namespace=namespace),
57            spec=client.V1LimitRangeSpec(
58                limits=[
59                    client.V1LimitRangeItem(
60                        type="Container",
61                        default={
62                            "cpu": lr_spec.default_cpu,
63                            "memory": lr_spec.default_memory,
64                        },
65                        default_request={
66                            "cpu": lr_spec.default_request_cpu,
67                            "memory": lr_spec.default_request_memory,
68                        },
69                    )
70                ]
71            ),
72        )
73        try:
74            self._api.replace_namespaced_limit_range("tenant-limits", namespace, lr_body)
75        except ApiException as exc:
76            if exc.status == 404:
77                self._api.create_namespaced_limit_range(namespace, lr_body)
78            else:
79                raise
80
81        return {"namespace": namespace, "tier": tier, "success": True}

Verify by running kubectl get namespace tenant-<your-id> --show-labels and confirming the platform/tenant-id and platform/tier labels are present, then run kubectl describe resourcequota tenant-quota -n tenant-<your-id> and kubectl describe limitrange tenant-limits -n tenant-<your-id> to confirm the hard limits match the values for the tier you provisioned.

Do's and Don'ts

Having walked through the provisioner implementation and seen how it fits into your discipline, the rules below distil the design decisions into actionable habits and the failure modes worth avoiding.

Do's

1. ✓Do define QuotaSpec and LimitRangeSpec as frozen=True dataclasses and centralize all tier values in the TIER_QUOTAS registry — frozen dataclasses prevent accidental mutation of quota values between the quota_for_tier lookup and the Kubernetes API call, and a single registry means a tier change (e.g., raising the enterprise GPU limit from 4 to 8) requires editing one dict, not hunting scattered YAML files or per-script literals.
2. ✓Do use replace_namespaced_resource_quota and replace_namespaced_limit_range as the primary call, falling back to the create_* variant only on a 404 — replace semantics make provision the single code path for both initial setup and tier changes; re-invoking with a new tier string atomically overwrites the existing quota object without requiring a separate update branch.
3. ✓Do catch ApiException with status 409 on namespace creation — and only 409 — treating a 409 as success makes the entire provision method safe to retry after a transient failure, while re-raising any other status ensures real errors like permission denials (403) or invalid specs (422) surface immediately rather than silently short-circuiting provisioning.

Don'ts

1. ✗Don't issue create_namespaced_resource_quota or create_namespaced_limit_range before the namespace create call completes — the Kubernetes API server rejects both ResourceQuota and LimitRange objects that reference a non-existent namespace, so the three calls must always execute in the strict order: namespace → quota → limit range.
2. ✗Don't bypass quota_for_tier by constructing quota bodies with inline string literals — quota_for_tier normalizes the tier string with .strip().lower() and raises ValueError on any unrecognized value (e.g., a misspelled "Enterprise" or an unsupported "premium"), failing fast before any API call; inline construction silently provisions the tenant with whatever ad-hoc limits were typed, leaving a misconfigured namespace with no error surfaced.
3. ✗Don't use the tenant_id value as the full namespace name without the tenant- prefix — the f"tenant-{tenant_id}" convention keeps tenant namespaces in a predictable, filterable group distinct from platform-internal namespaces; omitting the prefix risks collisions with names like default, kube-system, or other reserved Kubernetes namespaces that the API server may accept without warning.