Deploy evaluation platform with Helm and Grafana

Introduction

When you need to deploy shared evaluation infrastructure across model development teams, manually coordinating the evaluation API, worker pool, job queue, and result database is error-prone and hard to reproduce across environments. Helm charts give you a single, parameterized deployment unit that handles environment-specific scaling, persistent storage, and webhook integration from one source of truth. By the end of this lesson, you'll be able to author a Helm values generator that packages the full evaluation platform—API service, auto-scaling workers, Redis queue, PostgreSQL store, and model registry regression pipeline—and verify the deployment is healthy.

Key Terminology

• Helm values generator — A Python function (generate_eval_platform_values) that produces the complete chart configuration dictionary from a single environment parameter, centralizing all environment-specific branching—replica counts, autoscaling flags, resource limits—in one place rather than across separate override files.
• Sub-chart dependency — A third-party Helm chart (such as Bitnami Redis or PostgreSQL) declared as a child of the parent chart; setting "enabled": True in the values dict deploys the sub-chart alongside the primary components without a separate helm install call.
• External metric autoscaling — An HPA configuration that scales the worker replicaCount based on an application-specific metric (eval_queue_depth) rather than CPU utilization, so the worker pool grows in proportion to actual benchmark backlog pressure.
• Regression pipeline — The regressionPipeline values section that wires a registryWebhookSecret into the evaluation platform, causing every new model version registered in the model registry to automatically trigger a full benchmark run.
• helm upgrade --install — An idempotent Helm command that installs the chart if no release exists and upgrades it in place if one does, making the same invocation safe for both initial provisioning and subsequent rollouts.
• Post-deploy verification coroutine — The verify_eval_platform async function that queries /health and /api/v1/workers/status immediately after a release and returns a structured dict (api_healthy, workers_active, queue_depth, db_connected) suitable for CI smoke testing or post-deploy assertions.

Concepts

One Function as the Single Source of Truth

When a deployment spans four interconnected components—Evaluation API, Worker Pool, Redis job queue, and PostgreSQL result store—the most common failure mode is configuration drift: production and staging diverge because environment-specific values are edited in multiple places. Helm addresses this with a values dictionary consumed by chart templates, but those values still need to come from somewhere consistent.

The lesson's approach is to centralize all environment branching inside a single Python function. Everything that differs between staging and production—replica counts, autoscaling enabled/disabled, resource limits—is derived from the single environment parameter. The chart receives one coherent dict; no separate override files, no per-environment branches to keep in sync. This makes generate_eval_platform_values the single source of truth for what each environment looks like (see Code Walkthrough).

Scaling on Queue Depth, Not CPU

Worker pods running benchmark evaluations present an unusual autoscaling problem: a worker blocked waiting on a remote model endpoint may appear idle to CPU-based metrics while the job queue grows unchecked. CPU autoscaling would leave the queue backed up until workers happen to spike; the platform would be slow to react to bursts of new model versions arriving in the registry.

The autoscaling.metrics block in the worker values uses an External metric type keyed to eval_queue_depth with an averageValue target of "5". This tells Kubernetes to maintain roughly one worker per five queued jobs, scaling the pool from minReplicas: 2 up to maxReplicas: 20 based on actual workload pressure. The worker pool reacts to demand as it builds in the queue—well before CPU utilization would produce any meaningful signal (see Code Walkthrough).

Verification as a Typed Diagnostic Contract

A Helm release completing without error doesn't guarantee that all components are reachable and correctly wired together. The verify_eval_platform coroutine treats post-deploy verification as an explicit contract: it queries both /health (which reports database connectivity) and /api/v1/workers/status (which reports active worker count and queue depth), and returns a structured dict rather than a bare boolean.

This structure matters because each key maps to a distinct failure mode. api_healthy: False points to the API pod or ingress; db_connected: False points to the PostgreSQL sub-chart or its existingSecret; workers_active: 0 points to the worker deployment or its autoscaler configuration. Returning a typed snapshot makes the result directly assertable in a CI smoke-test step and gives operators an immediate, scoped starting point for diagnosis when something is wrong (see Code Walkthrough).

Code Walkthrough

Now that you understand the evaluation platform's four-component architecture—Evaluation API, Worker Pool, Job Queue, and Result Store—the next step is expressing that architecture as Helm values and confirming the deployment is live.

The generate_eval_platform_values function assembles the complete chart values dictionary from a single environment parameter. The API section sets replica counts and injects connection strings for Redis and PostgreSQL. The worker section raises both the replica floor and resource limits, because benchmark runs are CPU- and memory-intensive, and enables autoscaling keyed to the eval_queue_depth external metric so workers scale with queue pressure rather than CPU. Redis and PostgreSQL are included as sub-chart dependencies, and the regression pipeline section wires in the model registry webhook so every new model version triggers a benchmark run automatically.

Code snippetpython
1def generate_eval_platform_values(environment: str) -> dict:
2    is_prod = environment == "production"
3    return {
4        "api": {
5            "replicaCount": 2 if is_prod else 1,
6            "image": {"repository": "gcr.io/project/eval-api", "tag": "latest"},
7            "resources": {
8                "requests": {"cpu": "250m", "memory": "512Mi"},
9                "limits": {"cpu": "500m", "memory": "1Gi"},
10            },
11            "env": {
12                "REDIS_URL": "redis://eval-redis:6379",
13                "DATABASE_URL": "postgresql://eval:$(DB_PASSWORD)@eval-postgresql:5432/evaluations",
14                "MODEL_REGISTRY_URL": "http://model-registry:8000",
15            },
16        },
17        "worker": {
18            "replicaCount": 5 if is_prod else 2,
19            "image": {"repository": "gcr.io/project/eval-worker", "tag": "latest"},
20            "resources": {
21                "requests": {"cpu": "500m", "memory": "1Gi"},
22                "limits": {"cpu": "2000m", "memory": "4Gi"},
23            },
24            "autoscaling": {
25                "enabled": is_prod,
26                "minReplicas": 2,
27                "maxReplicas": 20,
28                "metrics": [{
29                    "type": "External",
30                    "external": {
31                        "metric": {"name": "eval_queue_depth"},
32                        "target": {"type": "AverageValue", "averageValue": "5"},
33                    },
34                }],
35            },
36        },
37        "redis": {
38            "enabled": True,
39            "architecture": "standalone",
40            "resources": {"limits": {"memory": "256Mi"}},
41        },
42        "postgresql": {
43            "enabled": True,
44            "auth": {"existingSecret": "eval-db-creds"},
45            "primary": {"persistence": {"size": "50Gi"}},
46        },
47        "regressionPipeline": {
48            "enabled": True,
49            "registryWebhookSecret": "eval-webhook-secret",
50        },
51    }

After deploying with helm upgrade --install, the verify_eval_platform coroutine confirms the platform is healthy by querying the API's health and worker-status endpoints. It returns a structured snapshot—API health, active worker count, queue depth, and database connectivity—that you can assert against in a smoke test or a post-deploy CI step.

Code snippetpython
1import httpx
2
3async def verify_eval_platform(api_url: str) -> dict:
4    async with httpx.AsyncClient(timeout=10) as client:
5        health_resp = await client.get(f"{api_url}/health")
6        workers_resp = await client.get(f"{api_url}/api/v1/workers/status")
7
8    health = health_resp.json()
9    workers = workers_resp.json()
10
11    return {
12        "api_healthy": health_resp.status_code == 200,
13        "workers_active": workers.get("active_count", 0),
14        "queue_depth": workers.get("queue_depth", 0),
15        "db_connected": health.get("database") == "ok",
16    }

You'll know it works when verify_eval_platform returns api_healthy: True, a non-zero workers_active count, and db_connected: True within seconds of the Helm release completing.

Do's and Don'ts

Having just walked through how the values generator and verification coroutine deploy and validate the evaluation platform, the following guardrails capture the configuration choices that keep that deployment healthy across environments.

Do's

1. ✓Do key worker autoscaling to eval_queue_depth rather than CPU — benchmark workloads are bursty and CPU utilization lags queue pressure; using the External metric targeting averageValue: 5 lets the HPA respond to actual job backlog so workers scale before requests pile up, not after cores saturate.
2. ✓Do pass PostgreSQL credentials via existingSecret: eval-db-creds instead of plaintext values — Helm values files are routinely stored in version control; a secret reference keeps the credential out of the chart artifact entirely and lets Kubernetes manage rotation without a re-deploy.
3. ✓Do assert verify_eval_platform returns api_healthy: True, a non-zero workers_active, and db_connected: True as your post-deploy smoke test — the coroutine queries both /health and /api/v1/workers/status independently, so a partial failure (API up, workers not registering) is caught before the release is marked successful in CI.

Don'ts

1. ✗Don't set worker resource limits equal to requests (e.g., cpu: 500m/cpu: 500m) — benchmark runs spike to the 2000m CPU and 4Gi memory ceiling defined in the walkthrough; undersized limits cause OOMKill mid-evaluation and silently fail the job rather than returning an error the regression pipeline can surface.
2. ✗Don't omit the regressionPipeline.registryWebhookSecret field when enabling the regression pipeline — without the eval-webhook-secret reference the model registry webhook has no way to authenticate incoming model-version events, meaning new model pushes will never trigger benchmark runs even though regressionPipeline.enabled: True is set.
3. ✗Don't reuse production replica counts (replicaCount: 2 for API, 5 for workers) in non-production environments — generate_eval_platform_values gates these on is_prod deliberately; flattening the environment check wastes cluster resources in staging and masks scaling bugs that only appear when the autoscaler (minReplicas: 2, maxReplicas: 20) is actually active.