Build agentic RAG with query decomposition and self-verification

Introduction

When you point a single-shot RAG pipeline at a multi-part question like "Which suppliers in our network have both ISO 27001 certification and active contracts with subsidiaries flagged for late payments?", it fires one vector search, returns a noisy mix of partial matches, and the answerer hallucinates the connections it can't actually retrieve — and the user has no signal that the answer is wrong. Agentic RAG fixes this by decomposing the question into atomic sub-queries, retrieving evidence for each, and self-verifying the assembled evidence before answering — looping back to retrieve more when confidence is too low. By the end of this lesson you'll be able to build an agentic RAG pipeline that decomposes complex queries into ordered sub-queries with dependency edges, retrieves evidence per sub-query, and self-verifies with a confidence-thresholded loop before producing a final answer.

Key Terminology

• Query decomposition — splitting a complex question into a list of atomic sub-queries with optional dependency edges, each retrievable in a single RAG call; it matters here because single-shot retrieval cannot satisfy multi-part questions.
• Self-verification — an LLM-driven check that compares the assembled evidence against the original question and decides whether retrieval is sufficient or must repeat; it matters here because it is the gate that turns retrieval into a loop instead of a one-shot.
• Follow-up query — a targeted sub-query emitted by the verifier when evidence is missing or contradictory, fed back into the retrieval loop; it matters here because it converts "not confident" into a concrete next retrieval instead of a refusal.
• Confidence threshold — the minimum verifier confidence that ends the loop and lets the agent answer; it matters here because it bounds the retrieval/verification loop and trades latency for grounding.

Concepts

Why single-shot retrieval breaks on multi-part questions

A single retrieval call ranks chunks by similarity to one embedded query. When the question carries multiple unrelated constraints ("ISO 27001 certified" AND "active contracts" AND "subsidiaries flagged for late payments"), no chunk satisfies all of them and the top-k results are a noisy mix of partial matches. The fix is not a larger k — it is one retrieval per constraint, then assembling the results.

Decomposition with dependency edges

The decomposer turns a compound question into atomic sub-queries with depends_on edges that order their execution. Independent sub-queries fan out in parallel; dependent sub-queries wait for their predecessors so they can substitute resolved values into their query text. That is what turns "find subsidiaries flagged for late payments, then find their parent suppliers" into two ordered retrievals instead of one impossible one (see Code Walkthrough).

The verify-and-loop control flow

After all sub-queries retrieve evidence, the verifier reads the original question and the assembled evidence and returns sufficient, confidence, missing_info, and follow_up_queries. If confidence is below min_confidence, the agent retrieves the follow-ups and re-verifies. A max-iteration cap stops runaway loops. This is the agentic part — the pipeline decides at runtime whether it has enough to answer.

Code Walkthrough

The two snippets below implement the decomposer and the verifier from the previous section. Together they cover query decomposition, dependency tracking, and the confidence-thresholded verification loop.

Query Decomposition Agent

QueryDecomposer uses an LLM, schema-constrained by Instructor, to split a complex query into atomic sub-queries with dependency tracking. SubQuery.depends_on carries the integer indexes of prerequisite sub-queries; the executor uses this to schedule independent calls in parallel and dependent calls in order. The Pydantic schema prevents malformed decompositions from reaching the executor.

Code snippetpython
1import instructor, openai
2from pydantic import BaseModel, Field
3
4class SubQuery(BaseModel):
5    query: str
6    reasoning: str
7    depends_on: list[int] = Field(default_factory=list)
8
9class DecomposedQuery(BaseModel):
10    original_query: str
11    sub_queries: list[SubQuery]
12
13class QueryDecomposer:
14    def __init__(self, model: str = "gpt-4o-mini"):
15        self.client = instructor.from_openai(openai.AsyncOpenAI())
16        self.model = model
17
18    async def decompose(self, query: str) -> DecomposedQuery:
19        result = await self.client.chat.completions.create(
20            model=self.model,
21            response_model=DecomposedQuery,
22            messages=[
23                {
24                    "role": "system",
25                    "content": (
26                        "Decompose the query into atomic sub-queries. "
27                        "Each sub-query targets one piece of information. "
28                        "Use depends_on (indexes) when a sub-query needs "
29                        "the result of an earlier one."
30                    ),
31                },
32                {"role": "user", "content": query},
33            ],
34        )
35        result.original_query = query
36        return result

Self-Verification Loop

SelfVerifier evaluates the assembled evidence against the original query and returns a structured verdict. The agentic_rag glue function decomposes once, retrieves per sub-query, then loops verify→follow-up→retrieve until the verifier returns sufficient=True above the confidence threshold or max_iters is hit.

Code snippetpython
1class VerificationResult(BaseModel):
2    sufficient: bool
3    confidence: float = Field(ge=0.0, le=1.0)
4    missing_info: list[str] = Field(default_factory=list)
5    follow_up_queries: list[str] = Field(default_factory=list)
6
7class SelfVerifier:
8    def __init__(self, model: str = "gpt-4o-mini", min_confidence: float = 0.7):
9        self.client = instructor.from_openai(openai.AsyncOpenAI())
10        self.model = model
11        self.min_confidence = min_confidence
12
13    async def verify(self, original_query: str, evidence: list[str]) -> VerificationResult:
14        evidence_text = "\n".join(f"- {e}" for e in evidence)
15        return await self.client.chat.completions.create(
16            model=self.model,
17            response_model=VerificationResult,
18            messages=[
19                {
20                    "role": "system",
21                    "content": (
22                        "Evaluate whether the evidence is sufficient to "
23                        "answer the query. If not, list what is missing "
24                        "and propose targeted follow-up queries."
25                    ),
26                },
27                {
28                    "role": "user",
29                    "content": f"Query: {original_query}\n\nEvidence:\n{evidence_text}",
30                },
31            ],
32        )
33
34async def agentic_rag(query: str, retrieve, max_iters: int = 3) -> list[str]:
35    decomposer, verifier = QueryDecomposer(), SelfVerifier()
36    decomposition = await decomposer.decompose(query)
37    evidence = [await retrieve(sq.query) for sq in decomposition.sub_queries]
38    for _ in range(max_iters):
39        result = await verifier.verify(query, evidence)
40        if result.sufficient and result.confidence >= verifier.min_confidence:
41            break
42        evidence.extend([await retrieve(q) for q in result.follow_up_queries])
43    return evidence

You'll know it works when a multi-constraint query that previously returned partial matches now produces a DecomposedQuery with one sub-query per constraint, the verifier returns sufficient=True above min_confidence within max_iters, and the final evidence list covers every constraint named in the original question.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do force atomic sub-queries with explicit depends_on edges — the executor can then parallelize independent retrievals and serialize dependent ones instead of running everything in sequence.
2. ✓Do cap the loop with min_confidence and max_iters — unanswerable questions fail fast instead of looping forever on follow-up queries that never close the gap.
3. ✓Do log each decomposition and verification result — they are the audit trail that explains why the agent answered (or refused) and the data you need when tuning min_confidence.

Don'ts

1. ✗Don't skip the SelfVerifier step — multi-hop questions routinely return partial evidence on the first pass, and shipping that straight to the answerer is how hallucinations land in production.
2. ✗Don't set min_confidence to 1.0 — verifier scores are LLM-judged and noisy near the ceiling, so the loop will run until max_iters on questions that are already answered.
3. ✗Don't feed raw chunks straight to the verifier — deduplicate and trim per-sub-query evidence first, or the verifier wastes its context on near-duplicates and under-reports missing_info.