Diagnosing confidently wrong RAG

What it is

"Confidently wrong" is a retrieval-augmented generation system producing a fluent, well-formatted, cited-looking answer that is false. It is a distinct failure class from a system that is obviously broken, and it is dangerous precisely because every conventional signal is green: HTTP 200, latency normal, no exception, no error rate. The failure is in the content, and content is not something your existing monitoring measures.

The term is commonly conflated with "hallucination", which is imprecise enough to be useless as a diagnosis. A RAG system can produce a wrong answer in at least five mechanically different ways, and each has a different fix. The diagnostic skill being tested in an interview is whether you decompose before you prescribe.

The problem the diagnosis solves

Teams reach for the wrong lever. The reflexive fixes, in the order I usually see them attempted, are: raise k, switch embedding models, add "do not make things up" to the prompt, and buy a bigger model. All four can help; none of them is a diagnosis, and three of the four can make things worse. Raising k on a system with a groundedness problem adds distractors. Swapping embedding models without a golden set replaces one unmeasured quality with another.

The diagnosis exists to answer one question first: did retrieval fail, or did generation fail? Everything downstream branches on that.

Mechanics: the diagnostic sequence

Step 0. Reproduce with a full trace

You cannot debug what you cannot see. The trace for one request must contain: the raw query, the rewritten or expanded query if any, every retrieved chunk with its document ID, chunk ID, retrieval score and rank, the fused ranking if hybrid, the reranker's scores, the exact assembled prompt including ordering, the model ID and version, sampling parameters, and the raw completion.

If your system cannot produce that trace, stop and build it. Everything below is impossible without it, and "we could not reproduce it" is the most common reason these bugs stay open for months.

Step 1. The context injection test, which splits the problem in one move

Take the failing query. Manually place the known-correct passage into the context and re-run generation with everything else identical.

The answer becomes correct. Retrieval is the problem. The generator was fine; it never saw the evidence.
The answer is still wrong. Generation is the problem. The model had the evidence and did not use it, or used it incorrectly.

This one test converts an ambiguous complaint into a bounded investigation, and it takes about five minutes. It is the single most useful thing in this page.

Step 2a. If retrieval failed

Walk these in order, because they are ordered by how often they are the cause:

Vocabulary mismatch. The query uses the user's words and the document uses the organisation's words. "Parental leave" versus "family care absence policy". Dense retrieval is supposed to handle this and often does not, because embedding models are trained on general text and your corpus has jargon. Detection: run the query as pure BM25 and as pure dense, separately, and see which one finds the document. Fix: hybrid retrieval with reciprocal rank fusion, plus query expansion using a synonym list built from your own query logs.

Chunk boundary split the answer. The fact spans two chunks, so neither chunk scores well and neither contains the whole answer. Detection: look at whether the correct document appears in the results at any rank with a low score, or not at all. Fix: overlap, structure-aware chunking that respects headings and tables, parent-document retrieval (embed the chunk, return the parent section), or contextual chunking where each chunk is prefixed with a generated summary of its position in the document.

The filter ate it. Access control, tenant, date range or language filters applied to an approximate nearest-neighbour index cause the recall cliff: HNSW's graph traversal visits candidates and then discards those failing the filter, so a selective filter can leave you with far fewer than k results, or with the wrong ones entirely. Detection: run the same query with filters disabled. Fix: pre-filtering with a filtered-search-capable index, partitioning the index by the high-cardinality filter (one index per tenant, or per language), or raising ef_search substantially when filters are selective. Never post-filter a top-k result set: it both destroys recall and leaks the existence of documents the user cannot see.

Analyzer or language mismatch. The document is in French, indexed with the English analyzer, so stemming is wrong and BM25 scores are garbage. Common in multilingual corpora and almost invisible until you check.

Stale index. The source was updated; the index was not. Detection: compare the chunk's content against the live source. Fix: measure and alert on index lag as an SLI, exactly as you would projection lag in CQRS.

k is too small, or the reranker is truncating. The document is at rank 47 and you retrieve 20. Detection is trivial once you have the trace: retrieve 200 and see where it lands.

Step 2b. If generation failed

Position effects. Relevant content placed in the middle of a long context is recalled worse than content at the beginning or end. Liu et al. documented this "lost in the middle" effect across several models and both open and closed systems. Fix: order the context so the highest-ranked evidence is at the start and the instruction is at the end, and reduce the amount of context rather than maximising it.

Parametric prior conflict. The model was trained on a fact and your document contradicts it. Product prices, version numbers and dates are the usual casualties. Fix: an explicit instruction that retrieved context overrides prior knowledge, plus per-claim citation so the conflict is visible in the output.

No abstention path. Nothing in the prompt permits the model to say it does not know, so it produces the most plausible completion, which is a guess. Fix: make abstention an explicit, rewarded option, and gate on retrieval confidence: if the top reranker score is below a threshold, return "I could not find this" rather than calling the generator at all. Systems without an abstention path do not have a hallucination problem, they have a design problem.

Too much context. Beyond some point, added context reduces accuracy rather than increasing it. If you retrieve 50 chunks because you can, distractors outnumber evidence.

Citations are decorative. If the model generates a citation marker rather than selecting one, the citation is not evidence of grounding. Fix: verify citations programmatically after generation by checking that each cited chunk ID exists in the retrieved set and, better, that the claim's key entities appear in the cited chunk.

Step 3. Turn the anecdote into a measurement

Two separate metric families, and conflating them is the classic mistake:

Layer	Metric	What it answers
Retrieval	recall@k, NDCG@k, MRR	Did the evidence reach the context?
Generation	groundedness / faithfulness, answer relevance	Did the answer follow from the evidence?

A golden set of 100 to 500 hand-labelled question-and-source pairs, covering the head and the ugly tail, is the artifact that makes all of this measurable. Build it before building the system, and treat it as the regression gate: any change to chunking, embedding model, retriever, reranker or prompt runs the suite, and a regression beyond a threshold blocks the merge.

A worked example

An internal policy assistant answers "how many vacation days do contractors accrue?" with a confident "15 days per year". The correct answer, per the contractor handbook, is that contractors do not accrue vacation.

The trace shows five retrieved chunks, all from the employee handbook, top score 0.83. The contractor handbook does not appear at any rank.

Context injection test: pasting the contractor policy paragraph into the context produces the correct answer. So retrieval failed, and the generator is fine.

Why did retrieval fail? Pure BM25 for "contractor vacation" ranks the contractor handbook first. Pure dense ranks it 34th. The embedding model is pulling "vacation accrual" toward the employee policy, which is longer, more detailed and semantically denser on the topic; "contractor" is one token of signal against a paragraph of topical similarity. This is textbook vocabulary and specificity mismatch, and it is why hybrid retrieval exists.

The fix, and the measured effect on a 180-question golden set:

Add BM25 alongside dense, fuse with reciprocal rank fusion at $k=60$. Recall@5 moves from 0.71 to 0.88.
Add contextual chunking: prefix each chunk with a one-line generated description of the document and section it came from, so "contractor handbook, section 4, leave" is in the embedded text. Recall@5 to 0.93.
Add a cross-encoder reranker over the top 50. NDCG@5 improves; recall@5 is unchanged by construction, since reranking cannot add documents retrieval missed. Worth stating explicitly, because candidates often propose a reranker as a fix for a recall problem, and it is not one.
Add an abstention gate at a reranker score threshold, tuned on the golden set so that abstention costs at most 3 percent of answerable questions.

The numbers in this example are illustrative of the shape of such a fix, not measurements from a specific published system. The published data point worth citing is Anthropic's contextual retrieval write-up, which reported that adding generated context to each chunk reduced top-20 retrieval failure rate by about 35 percent, that combining contextual embeddings with contextual BM25 reduced it by about 49 percent, and that adding reranking on top brought the total reduction to about 67 percent. The ordering of those interventions matches the ordering above, which is the useful part.

Production evidence

Anthropic's contextual retrieval work is the cleanest public evidence that chunk-level context is a first-order lever, with the failure-rate reductions above.

Liu et al., "Lost in the Middle" (TACL 2024) measured the position effect across multiple models and showed a U-shaped accuracy curve against the position of relevant information in the context, which is why context ordering is a design decision rather than an implementation detail.

Ragas and TruLens are the two widely used open-source harnesses that implement the retrieval-versus-generation metric split described above, with faithfulness and answer-relevance scorers; both are worth naming because they show the split is standard practice rather than a personal framework.

Elastic and OpenSearch both document the reciprocal rank fusion formula $1/(k + \text{rank})$ with $k = 60$ as their hybrid fusion default, which is the same constant from Cormack, Clarke and Buettcher's original 2009 paper. Citing where the 60 comes from is a nice depth signal.

The debate

The credible alternative to all of this is fine-tuning instead of retrieval: train the model on your corpus so the knowledge is parametric. It is the right choice when the knowledge is stable, when you need the model to adopt a form or style rather than recall facts, and when latency budgets cannot afford a retrieval hop. It is the wrong choice for anything that changes weekly, anything that needs per-user access control (a fine-tuned model cannot forget one user's documents), and anything that needs citations.

The other alternative is long context instead of retrieval: put the whole corpus in the window. This works for small corpora and it is genuinely simpler. It stops working on cost (you pay for every token on every request, though prompt caching mitigates this substantially), on the position effect above, and on access control, which long context handles by not handling it.

My position: retrieval with hybrid search and an abstention gate is the default; fine-tuning is for form, not facts; long context is a legitimate answer below roughly a few hundred thousand tokens of stable, non-access-controlled corpus. The one thing I would not do is treat the three as competitors. Production systems use retrieval for recall, a fine-tuned or few-shot-prompted model for output form, and long context for the retrieved evidence.

Follow-up Q&A

"Your users say the answers are wrong. Where do you start?" With one specific failing query and its full trace, not with the aggregate. Then the context injection test to split retrieval from generation. Aggregate complaints are unactionable; a single reproducible trace is a bug.

"How do you evaluate this without labelled data?" Bootstrap the golden set from real query logs: take the 100 most frequent queries and the 50 that produced thumbs-down, and have subject-matter experts label the correct source document (not the correct answer, which is much more work). That gives you a retrieval evaluation immediately. For generation, LLM-as-judge on faithfulness against the retrieved context works reasonably because it is a comparison task rather than a knowledge task, but you must control for its known biases: position bias (randomise the order of compared answers), verbosity bias (longer answers score higher, so normalise or instruct against it) and self-preference bias (a model favours its own outputs, so use a different model as the judge).

"When is a reranker the wrong fix?" When your problem is recall. A reranker reorders what retrieval returned; it cannot conjure a document that was never retrieved. If recall@50 is 0.6, your ceiling after reranking is 0.6. Fix the first-stage retriever first, then rerank to improve precision at small k.

"How do you prevent this class of bug from reaching production again?" An eval suite in CI that runs on every change to prompts, models, chunking or retrieval configuration, gating merge on regression beyond a threshold; version pinning on the model, because a provider updating a model underneath you is a silent behaviour change; and production monitoring of groundedness on a sample of live traffic, because the golden set is a fixed distribution and production is not.

"What is the difference between the retrieval failing and the model ignoring retrieval?" Retrieval failure means the evidence is not in the context; groundedness failure means it is there and the answer contradicts it or is not supported by it. They are measured differently (recall@k versus faithfulness), fixed differently (retrieval pipeline versus prompt, ordering and abstention), and owned differently. Conflating them is the reason teams spend a quarter swapping embedding models to fix a prompt bug.

Common misconceptions

The biggest is that hallucination is a model problem to be solved by a better model. In a RAG system the majority of confidently wrong answers I have traced were retrieval failures: the model behaved reasonably given context that did not contain the answer. Upgrading the model makes those answers more fluent and equally wrong.

The second is that citations prove grounding. If the model writes the citation rather than selecting it from a structured list, the citation is generated text with the same reliability as the rest of the generated text. Verify citations programmatically or do not claim them as a safeguard.

The third is that a higher k is safer. More context means more distractors and a worse position effect. Precision at small k beats recall at large k once you are past the point where the evidence is present.

Interview delivery note

Say this: "First I get a full trace for one failing query. Then I do the context injection test: paste the known-correct passage into the context and re-run. If the answer becomes right, it is a retrieval bug; if it stays wrong, it is a grounding bug. Those two have completely different fixes, and the most expensive mistake is treating a retrieval bug as a hallucination problem."

The depth signal is separating retrieval evaluation from generation evaluation and naming the metrics for each, plus mentioning the abstention gate. Candidates who have shipped RAG talk about recall@k and faithfulness as different numbers owned by different parts of the system. Candidates who have not talk about "accuracy".

The Staff & Team Lead Interview Reference