Implement Eval Gates in Argo Workflows That Block Promotion on Failure

Introduction

When a model update ships past a quality regression, users encounter degraded responses before any alert fires — and by then the bad artifact is already in production. Eval gates solve this by inserting an automated quality checkpoint directly into the promotion pipeline: if the artifact fails, it never reaches the target environment. By the end of this lesson, you will be able to build an Argo Workflows DAG that runs Promptfoo evaluation suites in parallel, aggregates their pass rates, and blocks promotion automatically when any suite scores below a configurable threshold.

Key Terminology

• Eval Gate — an automated quality checkpoint inserted into a promotion pipeline that blocks an artifact from advancing to the next environment when any evaluation suite scores below a configured pass rate.
• WorkflowTemplate — a reusable, parameterized Argo Workflows resource (kind: WorkflowTemplate) that defines the full eval-gate DAG once and accepts runtime parameters such as artifact-id, target-env, and pass-threshold at invocation time.
• DAG Task Dependency — a directed dependency declared in Argo's dag.tasks[].dependencies list that prevents a task from starting until all listed upstream tasks have completed; used here to ensure gate-decision waits for every concurrent eval suite before aggregating results.
• outputs.parameters / valueFrom.path — the Argo mechanism by which a container writes results to a file (e.g., /tmp/results.json) and the workflow captures that file's contents as a named parameter referenceable downstream via {{tasks.<task-name>.outputs.result}}.
• Pass Rate — the ratio successes / (successes + failures) computed from Promptfoo's results.stats block; the core metric compared against pass-threshold in make_gate_decision to determine whether each suite clears the gate.
• gate-decision Task — the terminal DAG task that receives every suite's result as a workflow parameter, calls make_gate_decision to evaluate the AND policy across all suites, and emits either a promote signal or a per-suite failure report that halts promotion.

Concepts

The Eval Gate Pattern

A model artifact in a GenAI pipeline does not have a binary correct/incorrect verdict the way compiled code does — it has a quality distribution measurable only across a suite of LLM-judge or deterministic checks. The eval gate pattern addresses the feedback-loop problem: rather than discovering regressions from production alerts (which fire after users are already affected), the gate makes quality a hard precondition of promotion. If the artifact does not score above the threshold on faithfulness, format compliance, safety, or relevance, the workflow halts at the gate and the artifact never reaches the target environment.

The key design decision is making the quality bar a runtime parameter (pass-threshold, defaulting to 0.90) rather than baking it into the workflow definition. Different artifact types, target environments, or business risk tolerances may require different thresholds. The same eval-gate-pipeline WorkflowTemplate serves all of them without modification.

DAG Topology: Fan-Out Then Fan-In

The three-phase DAG structure encodes two distinct constraints simultaneously. First, the four eval suites — faithfulness, format, safety, and relevance — are independent: none requires another suite's output to run. Declaring them as separate DAG tasks with no mutual dependencies lets Argo schedule all four concurrently, minimizing wall-clock latency. Second, the gate-decision task must not start until every suite has produced its result. Listing all four eval task names in gate-decision's dependencies array enforces this sequencing without any application-level synchronization code.

This fan-out / fan-in shape is what gives the pipeline both speed (parallel suites) and safety (no gate decision until all evidence is in). See Code Walkthrough for the full dag.tasks block that wires these relationships.

Result Propagation Across Task Boundaries

Each run-eval-suite container writes Promptfoo's structured JSON output to /tmp/results.json and exposes it through an outputs.parameters stanza using valueFrom.path. After the container exits, Argo reads that file and stores its contents as a named workflow parameter. The gate-decision task then references each suite's captured output via the {{tasks.<task-name>.outputs.result}} expression in its own arguments.parameters block.

This file-backed parameter pattern is how structured data crosses task boundaries in Argo without shared volumes or external storage. The container writes to a predictable path; the workflow layer handles lifting those bytes into the parameter graph so downstream tasks can consume them by name.

Threshold Aggregation and the Promote-or-Block Branch

The make_gate_decision function implements a strict AND policy: all_passed starts as True and flips to False the moment any single suite's pass_rate falls below threshold. This is intentional — a safety suite at 0.60 is not rescued by a faithfulness suite at 1.00. Each suite is an independent quality dimension, and a regression in any one dimension is a regression in the artifact as a whole.

The function returns promote: True or promote: False alongside a suite_outcomes dictionary that carries per-suite pass rates. This structured return value makes failures actionable: the caller immediately knows which suite failed and by how much, rather than receiving a bare boolean with no diagnostic context (see Code Walkthrough). When promote is False, the workflow can surface the suite_outcomes payload directly as a failure report, guiding the team toward the specific evaluation dimension that caused the block.

Code Walkthrough

Now that you understand the three-phase DAG structure — parallel eval suites, result aggregation, and the promote-or-block branch — here is the WorkflowTemplate that implements all three phases in a single deployable resource.

The eval-gate-pipeline WorkflowTemplate accepts four parameters (artifact-id, artifact-type, target-env, and pass-threshold) and uses a DAG to run the faithfulness, format, safety, and relevance suites concurrently. The gate-decision task declares dependencies on all four suite tasks, so it only starts once every suite has written its result.

Code snippetyaml
1apiVersion: argoproj.io/v1alpha1
2kind: WorkflowTemplate
3metadata:
4  name: eval-gate-pipeline
5spec:
6  entrypoint: eval-gate
7  arguments:
8    parameters:
9      - name: artifact-id
10      - name: artifact-type
11      - name: target-env
12      - name: pass-threshold
13        value: "0.90"
14  templates:
15    - name: eval-gate
16      dag:
17        tasks:
18          - name: eval-faithfulness
19            template: run-eval-suite
20            arguments:
21              parameters:
22                - name: suite
23                  value: faithfulness
24          - name: eval-format
25            template: run-eval-suite
26            arguments:
27              parameters:
28                - name: suite
29                  value: format
30          - name: eval-safety
31            template: run-eval-suite
32            arguments:
33              parameters:
34                - name: suite
35                  value: safety
36          - name: eval-relevance
37            template: run-eval-suite
38            arguments:
39              parameters:
40                - name: suite
41                  value: relevance
42          - name: gate-decision
43            template: make-decision
44            dependencies:
45              - eval-faithfulness
46              - eval-format
47              - eval-safety
48              - eval-relevance
49            arguments:
50              parameters:
51                - name: faithfulness-result
52                  value: "{{tasks.eval-faithfulness.outputs.result}}"
53                - name: format-result
54                  value: "{{tasks.eval-format.outputs.result}}"
55                - name: safety-result
56                  value: "{{tasks.eval-safety.outputs.result}}"
57                - name: relevance-result
58                  value: "{{tasks.eval-relevance.outputs.result}}"
59    - name: run-eval-suite
60      inputs:
61        parameters:
62          - name: suite
63      container:
64        image: ghcr.io/promptfoo/promptfoo:latest
65        command: ["sh", "-c"]
66        args:
67          - |
68            promptfoo eval \
69              -c /configs/eval-{{inputs.parameters.suite}}.yaml \
70              --output /tmp/results.json \
71              --no-progress-bar
72            cat /tmp/results.json
73        volumeMounts:
74          - name: eval-configs
75            mountPath: /configs
76      outputs:
77        parameters:
78          - name: result
79            valueFrom:
80              path: /tmp/results.json

The run-eval-suite template accepts the suite parameter, resolves the matching config from the mounted /configs volume, runs promptfoo eval, and writes structured output to /tmp/results.json. The outputs.parameters stanza captures that file so the downstream gate-decision task receives each suite's score as a workflow parameter.

The aggregation and branch logic lives in a Python function invoked by the make-decision template:

Code snippetpython
1import json
2
3def make_gate_decision(results: dict, threshold: float = 0.90) -> dict:
4    suite_outcomes = {}
5    all_passed = True
6
7    for suite_name, result_json in results.items():
8        data = json.loads(result_json) if isinstance(result_json, str) else result_json
9        stats = data.get("results", {}).get("stats", {})
10        total = stats.get("successes", 0) + stats.get("failures", 0)
11        passed = stats.get("successes", 0)
12        pass_rate = passed / total if total > 0 else 0.0
13        suite_outcomes[suite_name] = {"pass_rate": pass_rate, "passed": pass_rate >= threshold}
14        if pass_rate < threshold:
15            all_passed = False
16
17    return {"promote": all_passed, "suite_outcomes": suite_outcomes}

The function iterates each suite result, computes pass_rate from the Promptfoo stats block, and flips all_passed to False the moment any suite falls below threshold. A promote: True return triggers the deployment branch; promote: False emits the per-suite failure report and halts promotion entirely.

Confirm that calling make_gate_decision with a result payload where every suite reports 100% successes returns {"promote": True, ...}, and that introducing a single failing suite drops the response to {"promote": False, ...} with the offending suite's pass_rate below 0.90.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do declare dependencies on all four suite tasks (eval-faithfulness, eval-format, eval-safety, eval-relevance) in the gate-decision DAG task — without this, gate-decision can start before every run-eval-suite container has written its /tmp/results.json, causing make_gate_decision to receive empty or missing result payloads for one or more suites.
2. ✓Do capture each suite's Promptfoo output via outputs.parameters.valueFrom.path: /tmp/results.json — this is the only mechanism that threads each suite's structured result into the gate-decision task as a workflow parameter; omitting it means make_gate_decision has no data from that suite and cannot compute a pass_rate.
3. ✓Do implement the gate as a per-suite pass_rate < threshold check that flips all_passed to False on the first failing suite — this is the logic in make_gate_decision that prevents a 100% faithfulness score from masking a failed safety or relevance suite; the gate must block promotion when ANY individual suite misses the threshold, not only when the average does.

Don'ts

1. ✗Don't average pass rates across the four suites before comparing to pass-threshold — make_gate_decision intentionally checks each suite independently and halts promotion the moment any single pass_rate falls below threshold; averaging would allow a 0% safety pass rate to be hidden behind three perfect suites and let a dangerous artifact promote.
2. ✗Don't hardcode the threshold value inside make_gate_decision instead of threading it from the pass-threshold WorkflowTemplate parameter — the template exposes pass-threshold (defaulting to "0.90") precisely so different target environments can raise or lower the bar without editing the aggregation function or the WorkflowTemplate spec.
3. ✗Don't derive pass_rate from any field other than the results.stats.successes and results.stats.failures block in the Promptfoo JSON — make_gate_decision computes pass_rate = successes / (successes + failures) from that specific block; reading a top-level boolean or a different output field discards per-suite granularity and breaks the failure report that identifies which suite caused the gate to block.