Implement RAGAS metrics for RAG evaluation

Introduction

When you've shipped a Retrieval-Augmented Generation pipeline and it quietly returns wrong-but-plausible answers, a single end-to-end "quality" score can't tell you whether the retriever surfaced the wrong chunks or the generator hallucinated past correct ones — and without that signal, every fix is a guess and every regression ships invisibly. Teams that rely on opaque aggregate scoring routinely ship a 30% rise in hallucinations masked by a small improvement in fluency. By the end of this lesson you'll be able to instrument a RAG pipeline with the four RAGAS metrics — context precision, context recall, faithfulness, and answer relevancy — and read their joint signal to localize failures to the retriever or the generator.

Retrieval-Augmented Generation pipelines chain two fundamentally different subsystems—a retriever that surfaces context documents and a generator that synthesizes an answer—yet most teams evaluate only the final answer with a single "looks good" score. RAGAS (Retrieval Augmented Generation Assessment) decomposes RAG quality into four orthogonal metrics that isolate exactly where a pipeline breaks down: did the retriever surface the right context? Did the generator stay faithful to that context? Did the answer actually address the question? By instrumenting each metric independently, you gain actionable diagnostics rather than an opaque aggregate score.

This section walks through the RAGAS metric taxonomy, shows how each metric is computed at the LLM-judge level, and builds a complete evaluation harness that you can wire into a CI pipeline. Every code example targets RAGAS v0.2+ with OpenAI as the judge model, the same stack you will use in the accompanying lab where pgvector handles retrieval and GPT-4o handles generation.

Key Terminology

• Context Precision: A retrieval-plane metric measuring the ranking quality of retrieved contexts, computed as weighted precision-at-k where relevant contexts ranked higher receive greater weight.

• Context Recall: A retrieval-plane metric measuring the fraction of ground-truth claims attributable to at least one retrieved context, identifying information gaps the generator cannot compensate for.

• Faithfulness: A generation-plane metric measuring the fraction of claims in the generated answer that are verifiably supported by the retrieved contexts, directly quantifying hallucination rate.

• Answer Relevancy: A generation-plane metric measuring semantic alignment between the generated answer and the original question, computed via reverse question generation and embedding similarity.

• LLM-as-Judge: The evaluation paradigm where a language model (the judge) scores the outputs of another model (the generator), enabling automated evaluation at scale without human annotators for every sample.

• Evaluation Plane: The conceptual separation between retrieval-quality metrics (operating on contexts versus ground truth) and generation-quality metrics (operating on the answer versus contexts or question), enabling independent diagnosis of pipeline component failures.

Each of these metrics returns a float between 0.0 and 1.0, where 1.0 represents perfect performance. In production systems, teams typically set threshold gates—faithfulness above 0.90, context recall above 0.80—that must pass before a RAG configuration change is promoted to production. You will implement exactly this kind of threshold-based gating when you build the end-to-end evaluation pipeline in another goal and wire it into CI/CD with DeepEval in another goal.

Concepts

Interpreting metric interactions

Individual metric scores become far more valuable when you analyze them jointly. The following diagnostic patterns emerge consistently across production RAG systems:

• High context recall + low faithfulness: The retriever surfaces the right information, but the generator hallucinates beyond it. This indicates a prompt engineering problem—tighten the system prompt to instruct the model to answer only from the provided context, or reduce the generation temperature.

• Low context recall + high faithfulness: The retriever misses critical information, but the generator stays strictly within what it received. This is a retriever-side failure. Investigate your chunking strategy, embedding model, or top-k setting. Increasing k from 3 to 5 often recovers missing context at the cost of lower context precision.

• Low context precision + high context recall: The retriever finds all relevant information but buries it among irrelevant chunks. This wastes generator context window tokens and increases hallucination risk. Add a reranker (such as Cohere Rerank or a cross-encoder) between the retriever and generator to push relevant contexts to the top.

• Low answer relevancy + high faithfulness: The answer is well-grounded in context but does not address the user's actual question. This typically means the retrieved contexts themselves are off-topic—check that your query-to-embedding pipeline correctly interprets the user intent, or add a query rewriting step.

Code Walkthrough

The four-metric decomposition

Before writing any code, you need a precise mental model of what each RAGAS metric measures and which component of the RAG pipeline it indicts when the score drops.

The diagram below maps each RAGAS metric to the pipeline stage it measures, showing how context precision, context recall, faithfulness, and answer relevancy each indict a different component when their scores drop.

The diagram separates metrics into two evaluation planes. Context precision and context recall operate on the retrieval plane—they never look at the generated answer. Faithfulness and answer relevancy operate on the generation plane—they never look at the ground-truth answer directly (faithfulness checks context support, not correctness). This orthogonality is what makes RAGAS diagnostically powerful: a high-faithfulness, low-context-recall result means the generator is doing its job but the retriever is failing to surface the right documents.

• Context Precision: Measures whether the contexts that are actually relevant to the ground-truth answer appear near the top of the retrieved list rather than buried at position k. RAGAS computes this as a weighted precision-at-k where higher-ranked relevant contexts contribute more to the score. A score of 1.0 means every relevant context appears before every irrelevant one. Low context precision indicates a ranking problem in your retriever—the right documents exist in the corpus but your embedding similarity or reranker is not surfacing them first.

• Context Recall: Measures the fraction of claims in the ground-truth answer that can be attributed to at least one retrieved context. The LLM judge decomposes the ground truth into individual statements and checks whether each statement is supported by the retrieved contexts. A score of 0.6 means 40% of the ground-truth information is missing from the retrieved set entirely—no amount of generator improvement can recover it.

• Faithfulness: Measures the fraction of claims in the generated answer that are supported by the retrieved contexts. The judge first extracts atomic claims from the generated answer, then verifies each claim against the context set. A faithfulness score of 0.75 means one in four generated claims is a hallucination—the generator fabricated information not present in the provided context.

• Answer Relevancy: Measures how well the generated answer addresses the original question. RAGAS reverse-engineers this by generating synthetic questions from the answer and computing the cosine similarity between these synthetic questions and the original question embedding. A low score means the answer drifts off-topic or includes excessive irrelevant information, even if every claim in it is technically faithful to the context.

Preparing the evaluation dataset

RAGAS requires a specific data schema. Each evaluation sample must include the user question, the retrieved contexts as a list of strings, the generated answer, and optionally a ground-truth reference answer. Context precision and context recall require the ground truth; faithfulness and answer relevancy do not. This asymmetry matters in production because ground-truth labels are expensive—you can run faithfulness and answer relevancy monitoring on every request without human annotation.

The following code constructs an EvaluationDataset using the RAGAS SingleTurnSample class, which enforces the required schema and validates that all fields contain non-empty values. Each sample represents one question-answer interaction from your RAG pipeline, with the retrieved_contexts field capturing the exact text chunks your retriever returned and the reference field holding the human-annotated ground-truth answer that context recall and context precision will be measured against.

Code snippetpython
1from ragas import EvaluationDataset, SingleTurnSample
2
3# Each sample captures one full RAG interaction
4samples = [
5    SingleTurnSample(
6        user_input="What are the GDPR requirements for AI model training data?",
7        retrieved_contexts=[
8            "Article 6 of GDPR requires a lawful basis for processing personal data. "
9            "For AI training, legitimate interest (Art. 6(1)(f)) is commonly invoked, "
10            "but requires a balancing test against data subject rights.",
11            "The GDPR mandates data minimization (Art. 5(1)(c)), meaning only data "
12            "adequate, relevant, and limited to the purpose should be used in training.",
13            "Recital 47 notes that direct marketing may qualify as legitimate interest, "
14            "but this does not directly apply to AI model training scenarios.",
15        ],
16        response=(
17            "GDPR requires a lawful basis for using personal data in AI training, "
18            "typically legitimate interest under Article 6(1)(f), which demands a "
19            "balancing test. Data minimization under Article 5(1)(c) mandates that "
20            "training data be adequate, relevant, and limited to the stated purpose. "
21            "Organizations must also conduct a DPIA for high-risk processing."
22        ),
23        reference=(
24            "GDPR requires a lawful basis such as legitimate interest (Art. 6(1)(f)) "
25            "with a balancing test, data minimization (Art. 5(1)(c)), and a Data "
26            "Protection Impact Assessment (DPIA) under Article 35 for high-risk "
27            "AI training scenarios."
28        ),
29    ),
30    SingleTurnSample(
31        user_input="How does differential privacy protect training data?",
32        retrieved_contexts=[
33            "Differential privacy adds calibrated noise to query results or gradients "
34            "during training, bounding the influence any single record can have on "
35            "the trained model's parameters.",
36            "The privacy budget epsilon controls the privacy-utility tradeoff. "
37            "Smaller epsilon provides stronger guarantees but may degrade model accuracy.",
38        ],
39        response=(
40            "Differential privacy protects training data by adding calibrated noise "
41            "during model training, ensuring no single data record significantly "
42            "influences the final model. The epsilon parameter controls the tradeoff "
43            "between privacy strength and model utility."
44        ),
45        reference=(
46            "Differential privacy adds mathematical noise to bound per-record "
47            "influence on model parameters, controlled by the epsilon privacy budget "
48            "that governs the privacy-utility tradeoff."
49        ),
50    ),
51]
52
53dataset = EvaluationDataset(samples=samples)

• The imports bring in EvaluationDataset and SingleTurnSample from the top-level ragas module. EvaluationDataset is the container that the evaluate() function expects, while SingleTurnSample validates individual samples.
• The first SingleTurnSample is constructed with four required fields. The user_input is the original user query. The retrieved_contexts is a list of three context strings exactly as the retriever returned them—note the third context about "Recital 47" is partially irrelevant, which context precision will penalize. The response is the raw LLM-generated answer, and the reference is the human-annotated ground truth.
• That third entry in retrieved_contexts is deliberately semi-relevant. RAGAS context precision will detect that this chunk is ranked third but contributes minimally to the ground truth, reducing the precision score. This mirrors real-world retriever behavior where top-k results often include noise.
• The first sample's response includes a claim about DPIA that does not appear in any retrieved context. Faithfulness scoring will flag this as an unsupported claim, demonstrating how faithfulness catches hallucinations even when the claim is factually correct.
• The second SingleTurnSample demonstrates a cleaner retrieval scenario with two highly relevant contexts and no irrelevant ones, producing a natural baseline for comparison.
• The samples are wrapped into an EvaluationDataset that handles batching and validation internally.

Running the four-metric evaluation

With the dataset prepared, you now configure the RAGAS evaluator with all four metrics and an LLM judge. The evaluate function from the ragas module accepts a list of metric instances—ContextPrecision, ContextRecall, Faithfulness, and ResponseRelevancy—and an LLMFactory wrapper that tells RAGAS which model to use for the judge calls. Each metric makes multiple LLM calls per sample (claim extraction, claim verification, question generation), so you should expect 8-12 judge calls per sample when running all four metrics simultaneously.

Code snippetpython
1from ragas import evaluate
2from ragas.metrics import (
3    ContextPrecision,
4    ContextRecall,
5    Faithfulness,
6    ResponseRelevancy,
7)
8from ragas.llms import LangchainLLMWrapper
9from langchain_openai import ChatOpenAI
10
11# Configure the judge model — GPT-4o balances cost and judgment quality
12judge_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o", temperature=0))
13
14# Instantiate all four RAGAS metrics
15metrics = [
16    ContextPrecision(llm=judge_llm),
17    ContextRecall(llm=judge_llm),
18    Faithfulness(llm=judge_llm),
19    ResponseRelevancy(llm=judge_llm),
20]
21
22# Run evaluation across the full dataset
23result = evaluate(dataset=dataset, metrics=metrics)
24
25# Access the per-sample scores as a pandas DataFrame
26df = result.to_pandas()
27print(df[["user_input", "context_precision", "context_recall",
28          "faithfulness", "answer_relevancy"]].to_string(index=False))
29
30# Aggregate scores for pipeline-level reporting
31for metric_name in ["context_precision", "context_recall",
32                     "faithfulness", "answer_relevancy"]:
33    mean_score = df[metric_name].mean()
34    print(f"{metric_name}: {mean_score:.3f}")

• The imports bring in the four metric classes from ragas.metrics and the LangchainLLMWrapper adapter. RAGAS uses LangChain's model interface internally, so you wrap a ChatOpenAI instance. If you use Azure OpenAI, substitute AzureChatOpenAI here—the metric classes are model-agnostic.
• The judge model is configured with temperature=0. This is critical for evaluation reproducibility. A non-zero temperature introduces variance between runs, making it impossible to detect genuine metric movements from random judge fluctuation. Always use deterministic decoding for evaluation judges.
• Each metric is instantiated with the same judge LLM. You can use different models per metric—for example, a cheaper model for faithfulness (which makes many calls) and a stronger model for context recall (which requires nuanced reasoning)—but starting with a uniform judge simplifies debugging.
• The evaluate call orchestrates all judge calls, handling batching and retry logic internally. For large datasets (100+ samples), RAGAS parallelizes calls using async execution. It returns a Result object containing both aggregate scores and per-sample breakdowns.
• The result is converted to a pandas DataFrame where each row is a sample and each column is a metric score between 0.0 and 1.0. This is the format you will feed into statistical significance testing in another goal of this chapter.
• The mean score per metric is computed across all samples. In production dashboards, you would track these aggregates over time to detect regression. A faithfulness drop below 0.85 typically warrants immediate investigation—it means more than 15% of generated claims lack context support.

Do's and Don'ts

Do's

1. ✓Do read the four metrics as a joint diagnostic signal — pair the retrieval-plane scores (context_precision, context_recall) with the generation-plane scores (faithfulness, answer_relevancy) before drawing any conclusion; high faithfulness combined with low context_recall specifically implicates the retriever, because the generator is faithfully synthesizing whatever it received while 40%+ of the ground-truth information was never surfaced.
2. ✓Do run faithfulness and answer_relevancy on production traffic without ground-truth labels — RAGAS v0.2+ SingleTurnSample allows the reference field to be omitted for these two metrics, so you can monitor hallucinations and topic drift on every live request without incurring the cost of human annotation; reserve the labeled reference field for the context_precision and context_recall runs that genuinely require it.
3. ✓Do populate retrieved_contexts with the exact verbatim text chunks your retriever returned — the LLM judge decomposes both the generated answer (for faithfulness) and the ground-truth reference (for context recall) into atomic claims and verifies each claim directly against these strings, so passing summaries, document titles, or truncated snippets instead of full chunk text silently understates both scores.

Don'ts

1. ✗Don't collapse the four RAGAS scores into a single aggregate quality number — averaging context_precision, context_recall, faithfulness, and answer_relevancy destroys the orthogonality between the retrieval plane and the generation plane, reproducing exactly the opaque scoring problem RAGAS is designed to solve; a small fluency gain can mask a 30% rise in hallucinations when the scores are merged.
2. ✗Don't treat a low context_precision score as a corpus coverage gap — context precision is a weighted precision-at-k ranking metric that measures whether relevant chunks appear before irrelevant ones in the retrieved list; the right documents already exist in the corpus, and adding more data without fixing the embedding similarity or reranker will not raise the score.
3. ✗Don't omit the reference field when your evaluation goal includes context_recall — RAGAS's LLM judge decomposes the ground-truth reference into individual statements and checks each one against the retrieved contexts; without a non-empty reference, context recall cannot be computed, leaving retrieval coverage completely invisible and making it impossible to distinguish a retriever failure from a generator failure.