Why RAG retrieves relevant documents but still gives wrong answers

Start with one failed question and write down the facts a satisfactory answer must establish. Include the entity, product version, region, date, and requested exceptions when they change the answer. “The documents discuss exports” is a weaker finding than “the documents establish this plan's scheduling entitlement and retention period.”

A retriever selects candidates according to its matching method. Haystack's retriever documentation describes scoring and returning candidates, including approximate search in many dense retrieval implementations. That operation does not certify that the selected passages jointly establish every required fact. Haystack Retrievers

For the failing request, preserve four inspectable artifacts within the application's existing data-access rules:

• The question as asked, relevant conversation context, and the query actually used for retrieval.
• Retrieved passage text and IDs, source revisions, effective filters, ranks, and the passages retained after reranking.
• The final messages or rendered prompt sent to the model, including source labels and any trimming or summaries.
• The generated answer, its citations, and the model and prompt versions.

Record the authoritative passages that support the expected answer separately. A source URL alone is insufficient for comparison if its contents can change. This record is a proposed debugging method, not a claim that a tracing integration captures these fields automatically.

The distinction between evidence selection, faithful use, and response quality follows the original RAGAS paper. The boundary-by-boundary procedure below is editorial synthesis: it turns those separate questions into checks on a concrete request.

Use this decision tree when the answer is wrong despite plausible search results. At each step, inspect the evidence for the required facts, not just a document title or similarity score. If the answer to a check is unknown, collect that artifact before treating the check as passed.

1. Source coverage: Find current, authoritative evidence for each required fact in the corpus the application may use. If it exists in the original document, verify that ingestion preserved it in the searchable representation, including table headers and qualifiers. If it does not exist or access excludes it, choose a source update, an authorized source lookup, clarification, or an explicit statement of uncertainty.
2. Retrieval and ranking: Locate those passages in the results before and after ranking and filtering. If they are indexed and eligible but absent, inspect the query, filters, matching method, and cutoffs.
3. Context assembly: Compare the retained passages with the final model input. A correct search result can still lose its date, table header, exception, or entire paragraph during rendering, deduplication, summarization, or trimming.
4. Answer generation: If sufficient, applicable evidence reached the model, identify the exact unsupported, contradicted, or omitted answer claim. Investigate how the prompt and model used that evidence.

The first failed check selects the next investigation; it does not establish that downstream behavior is sound. Missing evidence and a model that guesses instead of acknowledging the gap can both need correction. After repairing a boundary, repeat the downstream checks.

Hypothetical example. AcmeFiles, its plans, documents, dates, and responses below are invented to illustrate diagnosis. No RAG system was run.

A user asks: “For an AcmeFiles Starter workspace in the EU on 19 September 2026, can I schedule daily exports, and how long are those exported files retained?”

Suppose the retrieved passages are:

An imagined answer, “Yes, schedule a daily export; the files are retained for 30 days,” answers the right topics but makes two unsupported moves: it assumes Starter eligibility and transfers a Pro retention rule to Starter. Adding citations to D2 and D3 would make those moves traceable, not valid.

Write the missing evidence explicitly:

With only D1–D3, an appropriate response would explain that the passages establish manual export and eligible-plan scheduling, but do not establish Starter EU eligibility or retention. It should seek the missing evidence if the application supports that lookup. The absence of an entitlement in these snippets does not establish that the feature is unavailable.

Now suppose two additional, authoritative fictional entries exist: E1 enables daily scheduling for Starter EU from 1 September 2026; E2 specifies seven-day retention for Starter EU scheduled exports. Assume neither has been superseded at the question's date. Together they support “Yes; seven days,” with a citation for each claim.

The next action depends on where E1 and E2 went. If they were never ingested, repair coverage. If indexed but missing from candidates, inspect retrieval. If selected but E2 was trimmed, inspect assembly. If both reached the model and it still chose Pro's 30-day rule, inspect evidence use. Changing chunk size is a candidate only when the trace shows a chunk boundary separating a needed fact from its scope.

The Ragas metric catalog contains separate measures because these questions need different evidence. Use the following distinctions when choosing a diagnostic; the rows are not interchangeable quality scores.

Definitions: Context Recall, Context Precision, Faithfulness, Factual Correctness, and Response Relevancy.

An answer can faithfully repeat a stale policy and still be wrong for today's question. It can also state a true fact from model knowledge without support in the retrieved passages. Those are different defects under a requirement for answers supported by current sources.

When a reference answer is unavailable, inspect answer-to-context support and collect the missing reference evidence. Do not substitute the system's own answer as an independent completeness standard: omitted facts would then disappear from both sides of the check. Ragas also documents context-recall variants using reference passages or passage IDs; select one only when those labels represent the evidence the task actually requires. Context Recall

This table gives proposed investigations, not guarantees about which configuration will win.

Haystack documents both combining retrieval methods and applying metadata filters; these are concrete examples of controls at the retrieval boundary. They do not repair facts absent from the source. Retrievers

Assembly deserves its own check. In Haystack documentation version 3.1, PromptBuilder renders supplied variables into a template. The documented default requires all template variables; if a variable is explicitly made optional and omitted, it is replaced with an empty string. A successful render therefore needs to be interpreted alongside that configuration, and even a required variable's presence does not prove its contents are sufficient. Inspect the actual rendered text. PromptBuilder

After finding a suspected boundary, test one change at a time against the captured failure. These are suggested comparisons for the reader to perform, not experiments reported by this guide.

The evidence-substitution comparison changes both content and potentially length, so a success alone does not isolate a retrieval algorithm. Likewise, one improved answer is not proof of a cause. Repeat generation comparisons enough to assess variability for the decision at hand, and retain the outputs rather than presenting a single favorable sample.

The 2023 study Lost in the Middle, v3 found position-dependent performance in its evaluated models and tasks. That motivates testing context placement; it does not establish the behavior of every model available in 2026 or a universal best ordering.

Turn the original failure into a regression case with required claims, applicable source passages, and acceptable handling of missing information. Add nearby cases that could expose a tradeoff: another plan, an earlier effective date, a superseded rule, a genuinely unanswerable question, and a question needing both documents. Keep answer correctness and unsupported-claim checks separate so that a system cannot appear improved merely by declining every question.

For each failed question, complete this checklist before selecting a broader retrieval or model change:

• Specify the required facts, scope, date, and acceptable uncertainty or clarification.
• Identify authoritative supporting passages, or record exactly which evidence is unavailable.
• Check that ingestion preserved those passages and their qualifiers in the allowed searchable corpus.
• Record the effective query and filters; locate each required passage before and after ranking.
• Compare selected passages with the exact final model input, including dates, labels, tables, summaries, and trimming.
• Map each disputed answer claim to supporting or contradicting text; record omitted requirements separately.
• Name the earliest observed failure and any additional downstream failure.
• Choose one targeted change and define what observation would support or reject the diagnosis.
• Retain a regression case and check both answerable and insufficient-evidence variants.

A completed record should produce a decision such as: “The retention passage was retrieved but omitted during prompt assembly; preserve that passage and verify the answer uses its seven-day rule.” “Improve RAG quality” is too broad to tell whether the chosen change addressed the failure.

Documentation context, checked 19 September 2026: Haystack pages identify documentation version 3.1. Ragas links use the moving latest documentation; the metric pages distinguish their collections API examples from legacy APIs. This guide uses their definitions, not a tested SDK installation. Before implementing a metric, record the installed Ragas version, exact metric variant, judge model, and reference-data revision so later comparisons mean the same thing. The 2023 RAGAS paper supplies the conceptual background, while current metric definitions come from the linked docs. Context Precision No SDK code, evaluation scores, or production measurements are reported here.