GenAI CI/CD Pipelines — chapter audio overview

Podcast Script: GenAI CI/CD Pipelines Host: Welcome back to GenAI Operations. This is Chapter 10 of 65, and today we're tackling CI/CD pipelines for GenAI artifacts. Now, if you're listening in the car, CI/CD stands for continuous integration and continuous delivery — it's the automated assembly line that takes a change from a developer's laptop all the way through testing and into production. This is a core competency for any team building production AI systems. Your organization invested in this training because they need engineers who don't just use AI tools, but who build the delivery infrastructure behind them. In the last chapter, we covered the GitOps control plane — treating Git as the source of truth and letting a controller sync that state to your cluster. Now the question is, how do things *get* into Git in a validated, trustworthy state? Picture this: your machine learning engineer tweaks a prompt to improve customer support replies. How do you make sure that one-word change doesn't double your hallucination rate in production? You'll practice this in six hands-on labs — but first, let's build the mental model. We'll explore why GenAI breaks traditional pipelines, the four-stage model, artifact-specific pipelines, and how to monitor the whole thing. Expert: Right, and let's start with why this is genuinely hard. Traditional software pipelines were built for a world where the thing you ship is a compiled binary or a container image. You compile it, you run unit tests, if it passes, you ship it. The failure modes are loud — a broken build throws a compiler error you can't miss. GenAI artifacts break that model completely. When we say GenAI artifact, we mean four specific things: a prompt template, which is the instruction text sent to a model with slots for variables; a model configuration, which is a file that defines settings like temperature and token limits; a RAG config — RAG stands for retrieval augmented generation, where the system fetches relevant documents before answering — and a guardrail policy, which is a safety rule file that defines what content gets blocked. None of these compile. A prompt can be perfectly valid text and still produce toxic or hallucinated output. A model config can have correct syntax and still blow up your cost budget by ten times. So we need pipelines that understand GenAI semantics, not just file syntax. The orchestration engine we use for this is called Argo Workflows. Think of Argo Workflows as a workflow engine that runs natively on Kubernetes — it lets you define a pipeline as a graph of steps where each step runs in its own isolated container pod. The critical piece is something called a WorkflowTemplate, which we'll just call a pipeline template. A pipeline template is a reusable, parameterized blueprint for a pipeline. You define it once — a skeleton that accepts inputs like which artifact to process and which environment to target — and then invoke it repeatedly with different parameters. So instead of maintaining sixty-five separate pipeline definitions for sixty-five prompts, you maintain one template and pass the prompt path as a parameter. The pipeline structure itself is a DAG — a directed acyclic graph — which is just a fancy way of saying a flowchart where steps can fan out to run in parallel and fan back in to merge results. This matters because evaluation is slow. If you can score a prompt against three different test datasets in parallel instead of one after another, your feedback loop drops from minutes to seconds. And because everything is a Kubernetes resource, you can store these pipeline templates in Git alongside the artifacts they validate, apply them with standard Kubernetes tooling, and query their status through the same APIs you already use. That tight integration with the cluster is why Argo beats older tools like Jenkins for this specific job. Host: Okay, so we have Argo Workflows orchestrating DAGs of containers, and pipeline templates that we parameterize rather than duplicate. That's the engine. Now the question becomes: what actually runs inside the pipeline? What are the stages, and what does each one check? Expert: The canonical GenAI pipeline has four stages, and each one enforces a different kind of contract. Stage one is lint. Linting is the fast structural check — think of it as proofreading before you submit an essay. The lint stage runs entirely offline, no network calls, no model calls. For a prompt template, it checks that every variable reference in the template has a matching entry in the variables list, that required metadata fields like version and author are present, and that the prompt doesn't exceed the model's context window when you fill in maximum-length test values. For a model config, it validates the file against a schema — which is a structured definition of what fields must exist and what values they can hold — using a tool called jsonschema. Lint catches the dumb stuff fast, within seconds, before you waste expensive compute on a broken file. Stage two is validate. Validation is where you start making network calls to confirm the artifact references real things in the real world. A model config might lint perfectly but reference a model endpoint that your organization doesn't actually have access to. Validation catches that by making a lightweight call to the provider's API to verify the model exists. For a RAG config, validation checks that the vector database collection exists and that the embedding model dimension matches the index dimension. Stage three is eval, short for evaluation. This is the most expensive and most important stage. Eval answers the question: does this artifact actually produce good outputs? You run the artifact against a curated test dataset — called a golden dataset because it's your ground truth — score the outputs against reference answers using metrics like semantic similarity or an LLM-as-judge rubric, and compare the scores against declared thresholds. A prompt might need to hit a mean score above 0.85 and a pass rate above 0.95. If it falls below either threshold, the stage fails and the pipeline halts. Stage four is promote. Promote doesn't evaluate anything — it moves a validated artifact from one environment to the next. Typically you have three environments: dev, staging, and production. Promotion enforces that you can only move forward through that chain, you can't skip staging, and production promotions require a recorded human approval. Every promotion writes an audit record with the eval scores, the pipeline run identifier, and who approved it. That provenance trail is gold when an incident happens in production — you can trace any deployed artifact back through its exact validation history to the commit that produced it. Now here's the key architectural point: these four stages are wired as a DAG where lint runs first, validate depends on lint, eval fans out into parallel runs against multiple datasets, and promote fans back in — it only proceeds once every eval branch passes. Argo handles the dependency resolution, the retry logic for infrastructure failures, and the artifact passing between stages automatically. Host: So lint is syntactic, validate is semantic against live infrastructure, eval is behavioral against golden datasets, and promote is the environment boundary. Four contracts, each catching a different class of defect. But you mentioned these pipelines need to behave differently for different artifact types. A prompt isn't the same as a RAG config. How do we handle that without rewriting the whole pipeline four times? Expert: The pattern is called a pipeline factory. You have a central dispatcher that inspects which files changed in a commit, matches the file path against a registry of prefixes, and routes to the correct artifact-specific pipeline. Files under a prompts directory go to the prompt pipeline. Files under a models directory go to the model config pipeline. Files under a rag directory go to the RAG pipeline. This path-based routing is critical because it prevents pipeline storms. Without it, a single commit touching fifty files spawns fifty redundant workflows that saturate your cluster. Let me walk through what makes each pipeline distinct. The prompt pipeline is the most frequently triggered — teams iterate on prompts daily. Its stages include a token estimation step, which expands the prompt template with maximum-length test values and counts how many tokens it consumes. A token, for context, is the atomic unit that language models process — roughly a short word or a word fragment. If the estimated count exceeds ninety percent of the model's context window, the stage fails. This catches a silent killer: a prompt that works fine in testing with short inputs but overflows the context window in production with long user queries. The prompt pipeline also runs an A/B evaluation stage that compares the new prompt against the current production baseline across a golden dataset, requiring a minimum score of 0.85 to pass. The model config pipeline is different. It adds a cost estimation stage, because changing one parameter — switching from a small model to a large one, or doubling the max tokens — can multiply inference cost by five to ten times. The cost estimation stage projects the new spend against current traffic and flags anything exceeding a twenty percent increase for manual approval. It also runs a canary deployment, which means routing five percent of live traffic through the new configuration while the other ninety-five percent stays on the current one, then comparing latency and error rates before full rollout. The RAG pipeline adds retrieval-specific checks. It validates the chunking parameters — chunking is how you split documents into pieces for the vector store, and the chunk size has to stay within empirical bounds, typically two hundred fifty-six to one thousand twenty-four tokens with ten to twenty-five percent overlap. It runs retrieval quality evaluation using metrics like recall at ten, which measures whether the correct document appears in the top ten retrieved results. And it runs a latency benchmark with a thousand queries to confirm the ninety-fifth percentile response time stays under the service level agreement, typically five hundred milliseconds. The guardrail policy pipeline is the most rigorous because it affects user safety directly. It runs a shadow-mode deployment stage where the new policy runs alongside the current production policy on live traffic samples, logging what each would block without actually blocking anything. A human then reviews the divergence report before the policy flips into enforcement mode. Skipping that shadow step is how you end up either over-blocking legitimate responses or under-blocking harmful ones. The beauty of the factory pattern is that all four pipelines share common building blocks — Git checkout, notification, promotion — but diverge in the stages that actually matter for their artifact type. Host: That's a clean separation — shared scaffolding, artifact-specific logic. So you've got pipelines humming along, processing artifacts all day. How do you know they're healthy? How do you catch the slow degradation — not a hard failure, but a gradual drift where evaluation scores slip from 0.92 down to 0.85 over a week? Expert: This is where pipeline observability becomes essential. Observability, in this context, means instrumenting every stage of every pipeline so you have continuous visibility into what's happening. There are four dimensions you need to track. The first is throughput — how many artifact versions move through the pipeline per hour, broken down by artifact type. A sudden drop signals a bottleneck. The second is evaluation score distributions. Raw pass/fail counts hide regression, because a score that slides from 0.92 to 0.85 over a week still technically passes an 0.80 threshold — but it's telling you something is degrading. Tracking the full distribution, not just pass/fail, exposes that drift. The third is stage latency profiles. A prompt lint step finishes in seconds, but a RAG retrieval evaluation can take minutes. Measuring the fiftieth, ninety-fifth, and ninety-ninth percentile latency per stage exposes infrastructure problems before they cause timeouts. The fourth is promotion conversion rate — the ratio of artifacts that enter the pipeline to those that reach production. A thirty percent rate for experimental guardrails is fine, but a thirty percent rate for routine prompt patches means your validation thresholds are misconfigured. The tooling stack for this has three layers. Argo Workflows has built-in exit handlers, which are hooks that fire when a step completes. Your exit handlers post structured events — stage name, artifact type, duration, status, evaluation score — to a metrics collector service. That collector exposes the data on a standard endpoint that Prometheus — an open-source time-series database designed for metrics — scrapes every fifteen seconds. Prometheus stores the data using four metric types. Counters increment monotonically, like total promotions. Gauges represent a current value that can go up or down, like the latest evaluation score. Histograms bucket observations into predefined ranges, which is how you compute latency percentiles without storing every individual measurement. Grafana — a dashboard tool that reads from Prometheus — visualizes the metrics in three dashboard rows: throughput on top, quality in the middle, latency heatmaps at the bottom. Finally, Alertmanager routes alerts when thresholds are breached. You want a warning alert when average evaluation score drops below 0.85 for fifteen minutes, giving operators time to investigate before it crosses the hard 0.80 threshold. You want a critical alert when the promotion rate falls below five percent for thirty minutes, which means the pipeline is effectively blocked. Every alert includes the artifact type in the payload, because the on-call responder needs to know immediately whether the problem affects prompts, model configs, RAG configs, or guardrails — each demands a completely different investigation path. Host: So observability isn't an afterthought — it's the nervous system that makes the whole pipeline trustworthy. Before we wrap up with the labs, give me the production wisdom. If I'm driving home tonight and I remember only two or three things from this chapter, what should they be? Expert: Three things. First: never promote a GenAI artifact on syntactic validation alone. A prompt can be perfectly valid YAML and still hallucinate. The single most common cause of GenAI production incidents traced back to CI/CD gaps is skipping behavioral evaluation. Always gate promotion on a quantitative threshold — a minimum score against a golden dataset. If you don't have a score, you don't have a pipeline, you have a publishing tool. Second: version every artifact independently with semantic versioning, and give each artifact type its own top-level pipeline. Don't build one monolithic workflow that handles prompts, model configs, RAG configs, and guardrails together. A schema change to the guardrail policy format will break the prompt promotion path, and you'll spend a Friday night debugging it. Separate pipelines, shared templates — that's the rule. Third: don't automatically retry failed evaluation stages. Evaluation failures in GenAI pipelines are almost never transient. They indicate a genuine regression, a corrupted dataset, or an upstream model behavior change. Configuring automatic retries on eval steps masks real problems and burns GPU compute on repeated failures. Retry only on infrastructure errors — out-of-memory kills, network timeouts — and treat a below-threshold score as terminal, routing it to a notification step that alerts the artifact owner with the full evaluation report. And the one absolute never: never pass secrets as plain-text pipeline parameters. API keys, vector database credentials, guardrail service tokens — these appear in the Argo UI, they get stored in workflow specs, they show up in controller logs. Always reference secrets through Kubernetes Secrets mounted as environment variables. Rotate them on a schedule. Your pipeline templates should reference secret names, never secret values. Host: That's the mental model. Now let's talk about what you'll build. You have six hands-on labs for this chapter. Lab one has you build a prompt CI/CD pipeline end-to-end. Lab two extends the pattern to model configuration and RAG configuration pipelines. Lab three implements pipeline observability with Prometheus metrics and Grafana dashboards. Lab four builds testing and validation for the pipelines themselves — testing your tests, essentially. Lab five focuses on performance optimization, reducing end-to-end pipeline latency. And lab six has you write the operational documentation that your team will actually use at three in the morning during an incident. Each lab has its own audio overview that goes deeper into the specifics. Host: Let's close with the key takeaways. You now understand why GenAI artifacts demand purpose-built pipelines — because prompts, model configs, RAG configs, and guardrail policies fail silently in ways that compilers and unit tests will never catch. You now understand the four canonical stages — lint, validate, eval, promote — and how they form a DAG in Argo Workflows where each stage enforces a different contract. And you now understand how pipeline observability closes the loop, turning opaque workflows into measurable systems with alerting on evaluation drift and promotion rate. You have the depth to evaluate CI/CD approaches for your team's GenAI platform and to explain the trade-offs in architecture discussions — why artifact-specific pipelines beat monolithic ones, why behavioral evaluation is non-negotiable, and why shadow mode matters for guardrails. The chapter quiz will focus on which stages catch which defects, how Argo Workflows and Events wire up triggering, what the GenAI-specific observability signals are, and how to select the correct pipeline during artifact promotion. Pay attention to the decision point between automatic and manual approval gates. In the next chapter, Chapter 11, we move to the GenAI Secret Manager — how to handle the API keys, model endpoints, and credentials these pipelines depend on without leaking them through workflow specs. It builds directly on the security posture we touched on today. Keep building.