Build DeepEval test suites for RAG

Introduction

When you ship a RAG pipeline without test-driven evaluation, every prompt tweak and retriever swap becomes a coin flip — quality drift slips through review and surfaces as user complaints in production. Teams that catch faithfulness or answer-relevancy regressions at merge time avoid the costly rollback-and-rerun cycle that follows a silent quality drop. By the end of this lesson you'll be able to build a DeepEval test suite that asserts minimum thresholds on faithfulness, answer relevancy, contextual precision, and contextual recall, and wire it into CI so failing scores block the merge.

Key Terminology

• LLMTestCase: The atomic unit of DeepEval evaluation that encapsulates a single query-response-context tuple — the input query, the RAG system's actual_output, an optional expected_output, and the retrieval_context the generation model consumed.
• Metric threshold: The minimum score (e.g. FaithfulnessMetric(threshold=0.7)) a test case must reach on a given quality dimension; any case scoring below it fails the test and blocks the merge.
• assert_test: The DeepEval function that bridges metrics into pytest, running an LLMTestCase against one or more metrics and raising an AssertionError if any score falls below its configured threshold.
• EvaluationDataset: The version-controlled collection of evaluation cases (loaded here from a JSON fixture) that, with stratified coverage and expert-curated ground truth, supplies the inputs, expected outputs, and retrieval contexts for the test suite.
• FaithfulnessMetric: An LLM-as-judge scorer that measures whether the generated answer is grounded in the supplied retrieval_context, asserting against a threshold (0.7 in this lesson) to catch ungrounded or hallucinated claims.

Concepts

Structuring Evaluation Datasets for Reliability

The quality of your DeepEval test suite depends entirely on the evaluation dataset. A poorly constructed fixture file produces unreliable metric scores that undermine confidence in your CI gate. Effective evaluation datasets follow specific structural patterns:

• Stratified coverage: Include test cases spanning different query types—factual lookups, multi-hop reasoning, comparison queries, and queries that require synthesizing information from multiple retrieved chunks. A dataset of 40–60 cases with balanced representation across query types provides sufficient signal for statistical significance testing
• Ground truth curation: For ContextualRecallMetric and AnswerRelevancyMetric with expected output, ground truth must be written by domain experts, not generated by the same LLM that produces the actual output. Using LLM-generated ground truth creates circular evaluation where the judge validates its own reasoning patterns
• Retrieval context capture: Each test case must include the actual retrieval_context your RAG pipeline returned, not idealized context. Record the retriever's output during dataset construction so that faithfulness and contextual precision metrics evaluate the real pipeline behavior, not a sanitized version
• Negative cases: Include queries where your RAG system should decline to answer or should indicate uncertainty. Test cases where expected_output is None or contains explicit hedging language verify that your system handles knowledge boundaries correctly

Calibrating Metric Thresholds

Now that you have your evaluation dataset structured and your metrics wired into pytest, the next step is choosing the threshold each metric asserts against. Calibrate empirically from a small set of known-good and known-bad cases rather than guessing. Take a handful of responses you have hand-labeled as clearly acceptable and a handful you know are bad — ungrounded answers, off-topic responses, weak retrievals — and run your metrics across both. A good threshold sits in the gap between the two distributions: high enough that every known-bad case fails, low enough that every known-good case passes. If the two groups overlap with no clean separating value, that is a signal the metric or the judge model needs attention before it can gate a build.

Building on that calibration, treat thresholds as values you tighten per environment rather than fix once. A pre-merge CI gate might run slightly looser thresholds to tolerate judge stochasticity and stay green on legitimate changes, while a release or production-promotion gate runs tighter ones. Keep every threshold in version control alongside the test suite — in the metric constructors or a small config file checked in next to your fixtures — so that any change to the quality bar is reviewed in the same pull request as the code it guards, and the bar that passed a given commit is always reproducible.

Code Walkthrough

DeepEval's Testing Philosophy

DeepEval models RAG evaluation as a first-class testing concern. Every evaluation scenario becomes an LLMTestCase—a structured object carrying the user query, the RAG system's actual output, the expected output (when available), and the retrieval context that the generation model consumed. Metrics are not standalone functions you call in a notebook; they are assertions that run inside pytest, producing pass/fail verdicts against configurable thresholds. This design means your RAG evaluation suite lives alongside your unit tests, integration tests, and end-to-end tests in the same repository, triggered by the same CI hooks.

The key architectural components form a layered pipeline:

• LLMTestCase: The atomic unit of evaluation, encapsulating a single query-response-context tuple along with optional expected outputs and retrieval contexts
• Metric: A scorer (e.g., FaithfulnessMetric, AnswerRelevancyMetric, ContextualPrecisionMetric, ContextualRecallMetric) that evaluates one quality dimension against a threshold
• Test Function: A pytest-decorated function that asserts one or more metrics against one or more test cases
• Test Suite: A collection of test functions organized by evaluation concern—retrieval quality, generation quality, or end-to-end RAG quality

The critical distinction from RAGAS is operational: RAGAS computes metrics and returns scores you interpret; DeepEval computes metrics and fails your build if scores drop below thresholds. Both use LLM-as-judge patterns internally, but DeepEval wraps the judgment in pytest's assertion machinery, making evaluation results actionable in automated pipelines.

This diagram illustrates the core integration pattern: DeepEval test suites run as a parallel test stage in CI, loading evaluation datasets, executing the RAG pipeline against each input, constructing LLMTestCase objects from the results, and asserting metric thresholds. Failures produce diagnostic output that includes per-metric scores and the LLM judge's reasoning, giving developers actionable feedback without leaving the CI dashboard.

Constructing Test Cases and Configuring Metrics

Building a DeepEval test suite starts with defining your evaluation dataset and wiring it into pytest-compatible test functions. The LLMTestCase class from deepeval.test_case requires at minimum an input (the user query) and an actual_output (the RAG system's response). For retrieval evaluation, you supply retrieval_context—the list of text chunks your retriever returned. For generation evaluation against ground truth, you supply expected_output. The assert_test function from deepeval runs a given LLMTestCase against one or more metric instances and raises an AssertionError if any metric score falls below its configured threshold. The following implementation demonstrates how to construct test cases using LLMTestCase, configure FaithfulnessMetric and AnswerRelevancyMetric with explicit thresholds, and wire them into a parametrized pytest function using @pytest.mark.parametrize for batch evaluation across a dataset loaded from a JSON fixture file.

Code snippetpython
1import json
2import pytest
3from deepeval import assert_test
4from deepeval.test_case import LLMTestCase
5from deepeval.metrics import (
6    FaithfulnessMetric,
7    AnswerRelevancyMetric,
8    ContextualPrecisionMetric,
9    ContextualRecallMetric,
10)
11
12def load_eval_dataset(path: str) -> list[dict]:
13    with open(path, "r") as f:
14        return json.load(f)
15
16EVAL_DATA = load_eval_dataset("tests/fixtures/rag_eval_cases.json")
17
18faithfulness = FaithfulnessMetric(
19    threshold=0.7,
20    model="gpt-4o",
21    include_reason=True,
22)
23answer_relevancy = AnswerRelevancyMetric(
24    threshold=0.7,
25    model="gpt-4o",
26    include_reason=True,
27)
28context_precision = ContextualPrecisionMetric(
29    threshold=0.6,
30    model="gpt-4o",
31    include_reason=True,
32)
33context_recall = ContextualRecallMetric(
34    threshold=0.6,
35    model="gpt-4o",
36    include_reason=True,
37)
38
39@pytest.mark.parametrize(
40    "case",
41    EVAL_DATA,
42    ids=[c["id"] for c in EVAL_DATA],
43)
44def test_rag_generation_quality(case: dict):
45    test_case = LLMTestCase(
46        input=case["query"],
47        actual_output=case["actual_output"],
48        expected_output=case.get("expected_output"),
49        retrieval_context=case["retrieval_context"],
50    )
51    assert_test(
52        test_case,
53        metrics=[
54            faithfulness,
55            answer_relevancy,
56            context_precision,
57            context_recall,
58        ],
59    )

• The imports bring in the core DeepEval components—assert_test is the assertion bridge between DeepEval metrics and pytest, LLMTestCase encapsulates each evaluation scenario, and the four metric classes cover both retrieval quality (contextual precision, contextual recall) and generation quality (faithfulness, answer relevancy).
• The load_eval_dataset helper reads evaluation cases from a JSON fixture file. Keeping test data in version-controlled fixtures ensures reproducibility across CI runs and enables dataset versioning alongside code changes.
• EVAL_DATA is loaded at module scope so pytest's parametrize decorator can iterate over cases during test collection, before any test function executes.
• FaithfulnessMetric is configured with a threshold of 0.7, meaning any test case scoring below 0.7 on faithfulness triggers a test failure. The model parameter specifies which LLM acts as the judge, and include_reason set to True ensures the judge's reasoning appears in failure output for debugging.
• AnswerRelevancyMetric uses the same threshold and judge model. This metric evaluates whether the generated answer directly addresses the user's query, independent of whether it is grounded in the retrieved context.
• The contextual metrics use a lower threshold of 0.6, reflecting that retrieval quality is often noisier than generation quality. ContextualPrecisionMetric measures what fraction of retrieved chunks are actually relevant, while ContextualRecallMetric measures what fraction of the ground-truth information appears in the retrieved chunks.
• The @pytest.mark.parametrize decorator creates one test invocation per evaluation case. The ids parameter uses each case's "id" field for readable test names in CI output—instead of test_rag_generation_quality[0], you see test_rag_generation_quality[query_about_transformers].
• Inside the test function, each dictionary from the fixture is converted to an LLMTestCase. The case.get("expected_output") call returns None when no ground truth is available, which is valid—faithfulness and answer relevancy do not require expected output. The assert_test call runs all four metrics and fails the test if any metric's score falls below its threshold.

CI/CD Integration and Evaluation Orchestration

Running DeepEval tests locally validates your metrics work. Running them in CI makes evaluation a gate on code quality. The integration pattern requires three considerations: managing the LLM judge API key as a CI secret, controlling evaluation cost by partitioning the dataset, and generating machine-readable reports that CI dashboards can parse. The following implementation creates a conftest.py file that configures DeepEval's evaluation model from environment variables, sets up a custom pytest plugin hook to capture per-metric scores into a JSON report artifact, and demonstrates how to use DeepEval's evaluate function for batch evaluation that produces aggregate statistics suitable for trend tracking across pipeline runs.

Code snippetpython
1import os
2import json
3import pytest
4from deepeval import evaluate
5from deepeval.test_case import LLMTestCase
6from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
7
8def build_test_cases(dataset: list[dict]) -> list[LLMTestCase]:
9    return [
10        LLMTestCase(
11            input=c["query"],
12            actual_output=c["actual_output"],
13            expected_output=c.get("expected_output"),
14            retrieval_context=c["retrieval_context"],
15        )
16        for c in dataset
17    ]
18
19def run_batch_evaluation(
20    test_cases: list[LLMTestCase],
21    output_path: str = "eval_results.json",
22) -> dict:
23    metrics = [
24        FaithfulnessMetric(threshold=0.7, model="gpt-4o"),
25        AnswerRelevancyMetric(threshold=0.7, model="gpt-4o"),
26    ]
27    results = evaluate(test_cases=test_cases, metrics=metrics)
28
29    summary = {
30        "total_cases": len(test_cases),
31        "passed": sum(1 for r in results.test_results if r.success),
32        "failed": sum(1 for r in results.test_results if not r.success),
33        "pass_rate": 0.0,
34        "metric_averages": {},
35    }
36    summary["pass_rate"] = summary["passed"] / max(summary["total_cases"], 1)
37
38    scores_by_metric: dict[str, list[float]] = {}
39    for test_result in results.test_results:
40        for metric_data in test_result.metrics_data:
41            name = metric_data.name
42            if name not in scores_by_metric:
43                scores_by_metric[name] = []
44            if metric_data.score is not None:
45                scores_by_metric[name].append(metric_data.score)
46
47    for name, scores in scores_by_metric.items():
48        summary["metric_averages"][name] = (
49            sum(scores) / len(scores) if scores else 0.0
50        )
51
52    with open(output_path, "w") as f:
53        json.dump(summary, f, indent=2)
54
55    return summary

• The imports combine standard library modules for environment and serialization with DeepEval's evaluate function—the batch evaluation entry point that runs multiple test cases against multiple metrics in a single call, producing an EvaluationResult object rather than individual pytest assertions.
• The build_test_cases factory function converts raw dictionaries into LLMTestCase instances using a list comprehension. Centralizing this conversion ensures consistent handling of optional fields—c.get("expected_output") returns None when the key is absent, which is valid for metrics that do not require ground truth.
• The run_batch_evaluation function accepts pre-built test cases and an output path for the JSON report artifact. Metrics are instantiated inside the function to keep threshold configuration co-located with the evaluation logic. The evaluate call returns an EvaluationResult containing per-case, per-metric scores.
• The summary dictionary tracks aggregate statistics. The pass_rate computation uses max(summary["total_cases"], 1) to avoid division by zero when the dataset is empty—a defensive pattern that prevents CI crashes on misconfigured fixtures.
• Score aggregation iterates over each test result's metrics_data list, grouping scores by metric name. The metric_data.score check against None filters out cases where a metric could not produce a score (e.g., the judge model returned an unparseable response).
• The per-metric averages are computed from the collected score lists. These averages enable trend tracking—when your CI system stores eval_results.json as a pipeline artifact, you can plot faithfulness and answer relevancy trends across commits.
• The JSON report is written to disk at the specified path. In CI, this file becomes a build artifact that downstream steps can consume—a Slack notification step might read pass_rate and post a summary, or a dashboard ingestion step might append the metric averages to a time-series database.

Do's and Don'ts

Do's

1. ✓Do version your evaluation dataset alongside your code - Store fixture files in tests/fixtures/ under version control so that dataset changes are reviewed in the same pull request as code changes, maintaining traceability between evaluation criteria and implementation.
2. ✓Do set include_reason=True on all metrics - The judge model's reasoning is the most valuable debugging artifact when a test fails. Without it, you know a test case scored 0.45 on faithfulness but not which claim was unsupported.
3. ✓Do run evaluation tests in a separate CI stage with longer timeouts - LLM-as-judge calls take 2–10 seconds each. A 50-case evaluation suite with four metrics makes 200 LLM calls. Set stage timeouts to 15–30 minutes and configure retry logic for transient API failures.

Don'ts

1. ✗Don't use the same LLM for both RAG generation and evaluation judging - If your RAG pipeline uses GPT-4o for generation, consider using a different model variant or provider for the judge to reduce systematic bias in faithfulness assessment.
2. ✗Don't set thresholds to 1.0 - Perfect scores are unreachable with LLM-as-judge metrics due to stochastic variation in judge responses. Thresholds of 0.95+ cause flaky tests that fail intermittently without reflecting actual quality changes.
3. ✗Don't skip retrieval context in test cases - Passing an empty retrieval_context list makes faithfulness evaluation meaningless. The metric cannot assess whether the output is grounded in retrieved information if no retrieval information is provided. Always capture and include the actual chunks your retriever returned.