Evaluate RAG quality with RAGAS metrics

Introduction

When you ship a RAG pipeline without quantitative quality signals, regressions hide in plain sight—an embedding model swap, a chunk-size tweak, or a prompt change can quietly halve faithfulness while every smoke test still passes. Teams that rely on spot-checks discover the drift only when a user surfaces a hallucinated citation in production. By the end of this lesson you'll be able to wire RAGAS metrics into your pgvector + BM25 hybrid pipeline, interpret faithfulness, answer relevancy, context precision, and context recall scores, and gate deployments on threshold-based pass/fail criteria.

Key Terminology

• Faithfulness: A RAGAS metric scoring whether every claim in the generated answer is supported by the retrieved contexts; low scores indicate hallucination.
• Answer Relevancy: A RAGAS metric measuring how directly the generated answer addresses the original user query, computed by generating synthetic questions from the answer and comparing them to the input query.
• Context Precision / Context Recall: Retrieval-stage RAGAS metrics — precision measures whether relevant chunks rank above irrelevant ones in the retrieved list, recall measures whether the retrieved chunks contain all information needed to answer the question (recall requires a ground-truth reference).

Concepts

Interpreting Scores and Diagnosing Failures

Raw RAGAS scores between 0.0 and 1.0 require calibration against your specific domain. A faithfulness score of 0.85 on legal documents may indicate dangerous hallucination rates, while the same score on casual Q&A might be acceptable. Establish baselines by running evaluate_batch against a golden test set of 50-100 annotated question-answer pairs before your first production deployment, then track deltas rather than absolute values.

When metrics drop, use the following diagnostic mapping:

• Low faithfulness (< 0.8): The LLM is generating claims not present in the retrieved contexts. Check your system prompt for instructions that encourage the model to "use its knowledge"—these override grounding. Also verify that your chunking from Unstructured or Crawl4AI preserves complete sentences; mid-sentence chunk boundaries force the LLM to hallucinate completions.
• Low answer relevancy (< 0.7): The answer drifts from the question. This often indicates over-retrieval—your hybrid search returns too many tangentially related chunks, and the LLM synthesizes a broad summary instead of a focused answer. Reduce top_k in your pgvector query or tighten the reciprocal rank fusion weights toward BM25 when queries contain specific keywords.
• Low context precision (< 0.7): Relevant chunks rank below irrelevant ones. This is a retrieval-stage failure. If your pgvector cosine similarity scores cluster tightly (all between 0.78-0.82), the embedding model lacks discriminative power for your domain—consider fine-tuning or switching to a domain-specific model. If BM25 scores dominate after fusion, your ts_vector configuration may need custom dictionaries for domain terminology.
• Low context recall (< 0.7): Retrieved contexts miss information needed to answer the question. This is a coverage failure—either your document ingestion pipeline skipped relevant sources, your chunk size is too small (splitting facts across boundaries), or your embedding model does not capture the query's semantic intent. Run the failing queries through retrieval alone and inspect which chunks are returned versus which ones contain the answer.

Connecting Evaluation to Your Agentic RAG Loop

The agentic RAG agent built with Pydantic AI in earlier sections iteratively retrieves, evaluates quality, and reformulates queries. RAGAS metrics formalize the "evaluate quality" step that the agent performs internally. Rather than relying on the LLM's self-assessment of answer quality (which is unreliable for factual grounding), you can inject a lightweight faithfulness check into the agent's decision loop. When the agent's internal evaluate_response tool returns False, it reformulates the query—but without RAGAS, "quality" is a vague heuristic. By computing faithfulness on the current response before deciding whether to reformulate, the agent gains a calibrated signal: if faithfulness drops below 0.8, reformulate; otherwise, return the answer.

This integration closes the loop between the four goals of this chapter. Crawl4AI and Unstructured ingest documents into pgvector. Hybrid retrieval with cosine similarity and BM25 fetches ranked contexts. LlamaIndex Workflows orchestrate the async pipeline steps. The agentic RAG layer reformulates queries when quality is low. And RAGAS provides the objective quality signal that drives the reformulation decision, replacing subjective LLM self-evaluation with decomposed, metric-driven assessment.

Code Walkthrough

Why RAG Evaluation Requires Specialized Metrics

Traditional NLP metrics like BLEU or ROUGE measure surface-level token overlap between a generated answer and a reference. They fail catastrophically for RAG because a factually correct answer can use entirely different wording than the ground truth, and a fluent answer can hallucinate facts not present in retrieved contexts. RAGAS addresses this by decomposing evaluation into retrieval quality (did you fetch the right chunks?) and generation quality (did the LLM use those chunks faithfully?). Each metric isolates a specific failure:

• Faithfulness catches hallucination—statements in the answer not supported by any retrieved context. A faithfulness score of 0.6 means 40% of claims in the generated response cannot be traced back to the provided chunks.
• Answer Relevancy detects tangential responses—the answer may be factually correct but fails to address the user's actual question. This metric generates synthetic questions from the answer and measures cosine similarity against the original query.
• Context Precision evaluates retrieval ranking—whether the relevant contexts appear near the top of the retrieved list. A precision of 1.0 means every relevant chunk ranks above every irrelevant one, directly measuring whether your reciprocal rank fusion from pgvector + BM25 produces well-ordered results.
• Context Recall measures retrieval coverage—whether the retrieved contexts contain all the information needed to answer the question. Low recall indicates your chunking strategy from Crawl4AI or Unstructured is splitting critical information across boundaries, or your embedding model fails to capture the query's semantic intent.

RAGAS evaluates your hybrid RAG pipeline across four orthogonal dimensions: Faithfulness checks whether the generated answer is grounded in the retrieved contexts, Answer Relevancy measures alignment between the answer and the original user query, Context Precision scores the ranking quality of chunks returned by pgvector + BM25 hybrid retrieval, and Context Recall compares retrieved context coverage against an optional ground truth answer. Together, these metrics pinpoint whether failures originate in retrieval, reranking, or generation.

Loading diagram...

The diagram above shows how each RAGAS metric taps into different parts of the pipeline. Faithfulness and answer relevancy evaluate the generation stage, while context precision and recall evaluate the retrieval stage. This separation lets you pinpoint whether a quality regression comes from your pgvector embeddings drifting, your BM25 weights being miscalibrated, or your LLM prompt template leaking hallucinations.

Core Evaluation Data Model

Before building the evaluator, you need a typed data structure that carries all four inputs required by RAGAS: the original query, the list of retrieved context strings, the generated answer, and an optional ground truth. The following code defines an EvalSample Pydantic model and the RAGEvaluator class with its evaluate_single method, which converts your pipeline outputs into a RAGAS EvaluationDataset and runs the four core metrics. The RAGEvaluator.init method accepts an optional LLMWrapper for the critic LLM that RAGAS uses internally to decompose claims and generate synthetic questions, defaulting to the same model your agentic RAG pipeline uses.

1from dataclasses import dataclass, field
2from ragas import EvaluationDataset, SingleTurnSample, evaluate
3from ragas.metrics import (
4    Faithfulness,
5    ResponseRelevancy,
6    LLMContextPrecisionWithoutReference,
7    LLMContextRecall,
8)
9from ragas.llms import LangchainLLMWrapper
10from langchain_openai import ChatOpenAI
11
12@dataclass
13class EvalSample:
14    query: str
15    contexts: list[str]
16    answer: str
17    ground_truth: str | None = None
18
19@dataclass
20class EvalResult:
21    faithfulness: float
22    answer_relevancy: float
23    context_precision: float
24    context_recall: float | None = None
25    raw_scores: dict = field(default_factory=dict)
26
27class RAGEvaluator:
28    def __init__(self, model_name: str = "gpt-4o-mini"):
29        self._critic_llm = LangchainLLMWrapper(
30            ChatOpenAI(model=model_name, temperature=0.0)
31        )
32        self._metrics_with_ref = [
33            Faithfulness(),
34            ResponseRelevancy(),
35            LLMContextPrecisionWithoutReference(),
36            LLMContextRecall(),
37        ]
38        self._metrics_no_ref = [
39            Faithfulness(),
40            ResponseRelevancy(),
41            LLMContextPrecisionWithoutReference(),
42        ]
43
44    def evaluate_single(self, sample: EvalSample) -> EvalResult:
45        has_ref = sample.ground_truth is not None
46        ragas_sample = SingleTurnSample(
47            user_input=sample.query,
48            retrieved_contexts=sample.contexts,
49            response=sample.answer,
50            reference=sample.ground_truth if has_ref else "",
51        )
52        dataset = EvaluationDataset(samples=[ragas_sample])
53        metrics = self._metrics_with_ref if has_ref else self._metrics_no_ref
54        result = evaluate(
55            dataset=dataset,
56            metrics=metrics,
57            llm=self._critic_llm,
58        )
59        scores = result.to_pandas().iloc[0].to_dict()
60        return EvalResult(
61            faithfulness=scores.get("faithfulness", 0.0),
62            answer_relevancy=scores.get("answer_relevancy", 0.0),
63            context_precision=scores.get("context_precision", 0.0),
64            context_recall=scores.get("context_recall") if has_ref else None,
65            raw_scores=scores,
66        )

Batch Evaluation with Pipeline Integration

Single-sample evaluation is useful for debugging, but production systems need batch evaluation across test sets to detect regressions. The following method extends RAGEvaluator with an evaluate_batch function that processes multiple samples, computes per-sample scores, and returns aggregate statistics. This integrates with your LlamaIndex Workflow outputs—after the SynthesizeStep produces a final answer and the RetrieveStep returns ranked contexts, you feed both into the evaluator. The method also supports threshold-based alerting, returning a boolean passed flag when all aggregate scores exceed configurable minimums.

1import statistics
2from dataclasses import dataclass
3
4@dataclass
5class BatchResult:
6    sample_results: list[EvalResult]
7    avg_faithfulness: float
8    avg_answer_relevancy: float
9    avg_context_precision: float
10    avg_context_recall: float | None
11    passed: bool
12
13# Add to RAGEvaluator class
14def evaluate_batch(
15    self,
16    samples: list[EvalSample],
17    thresholds: dict[str, float] | None = None,
18) -> BatchResult:
19    defaults = {
20        "faithfulness": 0.8,
21        "answer_relevancy": 0.7,
22        "context_precision": 0.7,
23        "context_recall": 0.7,
24    }
25    thresholds = thresholds or defaults
26    results = [self.evaluate_single(s) for s in samples]
27
28    avg_faith = statistics.mean(r.faithfulness for r in results)
29    avg_relevancy = statistics.mean(r.answer_relevancy for r in results)
30    avg_precision = statistics.mean(r.context_precision for r in results)
31
32    recall_scores = [r.context_recall for r in results if r.context_recall is not None]
33    avg_recall = statistics.mean(recall_scores) if recall_scores else None
34
35    passed = (
36        avg_faith >= thresholds["faithfulness"]
37        and avg_relevancy >= thresholds["answer_relevancy"]
38        and avg_precision >= thresholds["context_precision"]
39    )
40    if avg_recall is not None:
41        passed = passed and avg_recall >= thresholds["context_recall"]
42
43    return BatchResult(
44        sample_results=results,
45        avg_faithfulness=avg_faith,
46        avg_answer_relevancy=avg_relevancy,
47        avg_context_precision=avg_precision,
48        avg_context_recall=avg_recall,
49        passed=passed,
50    )

Do's and Don'ts

Do's

1. ✓Do read context precision and context recall before diagnosing faithfulness or answer relevancy — precision and recall expose failures in your pgvector + BM25 reciprocal rank fusion and Crawl4AI/Unstructured chunking strategy, while faithfulness and answer relevancy expose generation-layer failures; conflating the two leads you to rewrite prompts when the real problem is retrieval ranking or chunk-boundary splits.
2. ✓Do populate ground_truth in every EvalSample when retrieval coverage matters — LLMContextRecall silently returns None when ground_truth is absent, leaving chunk-boundary gaps and embedding-model semantic misses completely invisible at evaluation time.
3. ✓Do gate deployments on threshold-based pass/fail criteria across all four metrics — an embedding model swap, chunk-size tweak, or prompt change each degrade different RAGAS dimensions independently, and smoke tests cannot detect a faithfulness drop from 1.0 to 0.6 that leaves 40% of generated claims unsupported by retrieved contexts.

Don'ts

1. ✗Don't substitute BLEU or ROUGE for RAGAS when evaluating a RAG pipeline — surface token overlap fails catastrophically here because a factually correct answer may use entirely different wording than the reference, and a fluent hallucinated answer that matches reference phrasing scores high, inverting the quality signal you actually need.
2. ✗Don't treat a low context precision score as a prompt engineering or generation problem — a context precision below 1.0 means irrelevant chunks are outranking relevant ones in your pgvector + BM25 hybrid output, so the fix lives in reciprocal rank fusion weights or the reranker, not in the LLM generation stage.
3. ✗Don't interpret answer relevancy in isolation from faithfulness — a high answer relevancy paired with low faithfulness signals that the LLM is producing on-topic but hallucinated responses; RAGAS surfaces this compound failure only because each metric isolates a distinct orthogonal dimension of the pipeline.