Automate tenant onboarding with namespace provisioning and secret management

Introduction

When you onboard enterprise tenants by hand, you risk namespace collisions, unbounded resource consumption, and credentials that outlive a contract—each a service-availability incident waiting to happen. This lesson walks through automating the full provisioning pipeline: validating tenant configuration with a Pydantic schema, translating tier membership into Kubernetes ResourceQuota manifests, and exposing the workflow through a FastAPI endpoint that is safe to retry. By the end, you'll be able to build an onboarding service that creates an isolated namespace, enforces CPU and memory ceilings per tier, and stores provider secrets—turning error-prone manual steps into a repeatable, auditable API call.

Key Terminology

• TenantTier — A str enum with three levels (STARTER, BUSINESS, ENTERPRISE) that encodes a tenant's service tier and acts as the lookup key in TIER_QUOTAS, determining the CPU, memory, pod, and PVC ceilings applied to that tenant's Kubernetes namespace.
• Tenant model — A Pydantic BaseModel that serves as the first validation gate in the provisioning pipeline, enforcing a DNS-compatible regex on name, auto-generating a UUID tenant_id, and rejecting invalid budget_limit values before any cluster resource is touched.
• Kubernetes ResourceQuota — A cluster object of kind ResourceQuota that enforces hard ceilings on compute and storage within a single namespace; its spec.hard dict maps resource types such as cpu, memory, pods, and persistentvolumeclaims to their maximum allowed values.
• DNS label — A naming constraint that restricts Kubernetes namespace names to lowercase alphanumeric characters and hyphens starting with a letter; the Tenant.name regex pattern ^[a-z][a-z0-9-]*$ enforces this because the name is embedded directly in the namespace path tenant-{name}.
• TIER_QUOTAS — A dict mapping each TenantTier value to a hard resource-limit specification; STARTER is capped at 4 CPU and 8 Gi for development workloads, BUSINESS at 16 CPU and 32 Gi, and ENTERPRISE at 64 CPU and 128 Gi for production-scale GenAI inference.

Concepts

Schema Validation as the Provisioning Gate

When a provisioning request arrives, the first thing that must happen is schema enforcement — not cluster interaction. The Tenant Pydantic model serves this role: it validates name against a DNS label regex, confirms budget_limit is a positive number, ensures tier is one of the recognized TenantTier values, and auto-generates a tenant_id before any Kubernetes API call is issued. This ordering matters because cluster operations are expensive to roll back. A namespace created for a malformed tenant name will later fail quota application, and cleaning up partially-provisioned state is exactly the kind of error-prone work automation is meant to eliminate. By treating the Tenant model as the first gate, invalid tenants are rejected cheaply at the request boundary.

The DNS label constraint on name is not arbitrary. Since the namespace is constructed as tenant-{name}, the name must already be a valid Kubernetes namespace segment. The regex ^[a-z][a-z0-9-]*$ rejects uppercase letters, underscores, and leading digits at model instantiation time — long before the Kubernetes API server would reject them — turning a cluster-level error into a clear 422 at the API surface.

Translating Tier Membership into Resource Ceilings

The TIER_QUOTAS dict encodes a business decision about what each service tier is worth in compute terms. Each entry specifies hard limits not only on CPU and memory, but also on pod count and persistent volume claims — preventing a tenant from exhausting shared cluster nodes through pod proliferation even when staying within CPU ceilings. The key insight is that quota generation is intentionally a pure, stateless operation: build_resource_quota receives a validated Tenant, looks up tenant.tier in TIER_QUOTAS, and assembles a ResourceQuota manifest dict with the correct namespace field. It returns a plain Python dict that the onboarding endpoint can apply via the Python Kubernetes client or kubectl apply. Keeping this logic pure makes it independently testable against the expected manifest shape without touching a live cluster (see Code Walkthrough).

Namespace Isolation and Apply Ordering

A Kubernetes ResourceQuota is scoped to exactly one namespace, so limits defined in one tenant's quota manifest have no effect on any other namespace in the cluster. This namespace-per-tenant model is the primary isolation boundary: CPU, memory, pods, and PVCs consumed by one tenant are invisible to the quota enforcement of another, and a misbehaving tenant cannot borrow headroom from neighbors.

One critical sequencing constraint exists: the namespace must be created before the ResourceQuota manifest is applied. Attempting to apply a quota to a non-existent namespace is rejected by the Kubernetes API server. The onboarding pipeline must therefore treat namespace creation as a prerequisite step, and in a retry-safe endpoint — where the same request may be replayed — namespace creation should be idempotent (create-if-not-exists) and must precede the quota apply in every execution path. Skipping or reordering these steps surfaces as cryptic API server errors rather than clear validation failures, making sequencing discipline a correctness requirement, not just a style preference.

Code Walkthrough

With the Tenant model and TenantTier enum in hand, the implementation below shows how Pydantic validation, tier-based quota generation, and Kubernetes manifest construction compose into the core of the onboarding pipeline.

The Tenant model is the first gate every provisioning request must pass. It enforces DNS-compatible naming via a regex pattern—since the tenant name becomes the namespace suffix tenant-{name}—auto-generates a UUID-based tenant_id, and ties each tenant to a TenantTier that downstream quota logic depends on:

Code snippetpython
1from pydantic import BaseModel, Field
2from enum import Enum
3from typing import List
4from uuid import uuid4
5from datetime import datetime
6
7class TenantTier(str, Enum):
8    STARTER = "starter"
9    BUSINESS = "business"
10    ENTERPRISE = "enterprise"
11
12class Tenant(BaseModel):
13    tenant_id: str = Field(default_factory=lambda: str(uuid4()))
14    name: str = Field(..., min_length=3, max_length=63,
15                      pattern=r"^[a-z][a-z0-9-]*$")
16    tier: TenantTier
17    admin_email: str
18    provider_access: List[str] = Field(
19        default=["openai"],
20        description="Providers this tenant can access"
21    )
22    budget_limit: float = Field(
23        gt=0,
24        description="Monthly budget limit in USD"
25    )
26    created_at: datetime = Field(default_factory=datetime.utcnow)
27    status: str = Field(default="provisioning")

TenantTier is a string enum with three levels—STARTER, BUSINESS, ENTERPRISE—that map to progressively larger Kubernetes resource limits. The name field's regex ensures the value is a valid DNS label, preventing provisioning failures when the namespace is created in the cluster. The provider_access list controls which LLM backends the tenant may call, and budget_limit sets a monthly USD ceiling validated at instantiation so misconfigured tenants never enter the system.

Once a valid Tenant is constructed, build_resource_quota translates its tier into a Kubernetes ResourceQuota manifest ready to apply via the Python Kubernetes client or kubectl:

Code snippetpython
1TIER_QUOTAS = {
2    TenantTier.STARTER:    {"cpu": "4",  "memory": "8Gi",   "pods": "20",  "persistentvolumeclaims": "5"},
3    TenantTier.BUSINESS:   {"cpu": "16", "memory": "32Gi",  "pods": "50",  "persistentvolumeclaims": "20"},
4    TenantTier.ENTERPRISE: {"cpu": "64", "memory": "128Gi", "pods": "200", "persistentvolumeclaims": "50"},
5}
6
7def build_resource_quota(tenant: Tenant) -> dict:
8    quota = TIER_QUOTAS[tenant.tier]
9    return {
10        "apiVersion": "v1",
11        "kind": "ResourceQuota",
12        "metadata": {
13            "name": f"tenant-quota-{tenant.name}",
14            "namespace": f"tenant-{tenant.name}",
15        },
16        "spec": {"hard": quota},
17    }

TIER_QUOTAS maps each tier to hard limits on CPU cores, memory, running pods, and persistent volume claims. A STARTER tenant is capped at 4 CPU and 8 Gi—appropriate for development workloads—while an ENTERPRISE tenant receives 64 CPU and 128 Gi for production-scale GenAI inference. build_resource_quota looks up the tenant's tier, constructs the manifest with the correct namespace reference, and returns a plain dict that the onboarding endpoint applies to the cluster. The namespace field tenant-{tenant.name} must already exist (created by the preceding provisioning step) before the quota manifest is applied; applying a quota to a non-existent namespace will be rejected by the Kubernetes API server.

Verify by instantiating a Tenant with tier=TenantTier.STARTER, calling build_resource_quota, and confirming the returned dict's metadata["namespace"] equals "tenant-{name}" and spec["hard"]["cpu"] equals "4".

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do enforce DNS-compatible naming on Tenant.name via the pattern=r"^[a-z][a-z0-9-]*$" field constraint — this name becomes the namespace suffix tenant-{name}, and any uppercase letter, underscore, or leading digit will cause the Kubernetes API server to reject namespace creation before a single quota is applied.
2. ✓Do create the namespace as a distinct provisioning step before calling build_resource_quota and applying its manifest — the Kubernetes API server rejects a ResourceQuota applied to a non-existent namespace, leaving the tenant stuck in status="provisioning" with no resource ceiling in place.
3. ✓Do derive hard resource limits from TIER_QUOTAS keyed on TenantTier rather than accepting raw CPU or memory values from callers — this keeps the STARTER/BUSINESS/ENTERPRISE ceilings (e.g., 4 CPU/8 Gi vs. 64 CPU/128 Gi) server-authoritative and prevents a misconfigured or malicious request from self-assigning enterprise-scale quotas.

Don'ts

1. ✗Don't skip or defer the budget_limit gt=0 Pydantic validation by accepting a zero or negative value — tenants that bypass this field-level check at instantiation enter the provisioning pipeline without a monthly USD ceiling, creating unbounded cost exposure before any LLM backend calls are gated.
2. ✗Don't apply the ResourceQuota manifest in the same step that creates the namespace without verifying creation succeeded first — a race or partial failure leaves the namespace present but unquota'd, allowing a STARTER tenant to consume ENTERPRISE-level cluster resources until the next reconciliation.
3. ✗Don't hardcode a provider name directly into Tenant.provider_access at the call site instead of validating it against the allowed set — the provider_access list controls which LLM backends (e.g., "openai") the tenant may call, and an unvalidated string (e.g., "gpt4" or a typo) will produce a silently misconfigured tenant whose backend routing fails at request time rather than at onboarding.