Evaluating RAG: two systems, two metric families

What it is

A RAG system is two systems in a trench coat, and evaluating it means evaluating both separately.

Retrieval evaluation asks: did the evidence reach the context? Metrics are information-retrieval metrics over a labelled set: recall@k, NDCG@k, MRR, and increasingly context precision (how much of what you retrieved was actually relevant).

Generation evaluation asks: given the context, was the answer right? Metrics are groundedness (also called faithfulness: is every claim supported by the retrieved context), answer relevance (does it address the question), and correctness against a reference where one exists.

The single most common mistake in this area is collapsing them into one number called "accuracy". That number cannot tell you whether to fix the retriever or the prompt, which means it cannot direct any work, which means it is not a metric, it is a mood.

It is also commonly confused with model evaluation. Benchmark scores for the underlying model tell you almost nothing about your RAG system, because the model is the part you did not build and cannot change.

The problem it solves

Without the split, every quality regression turns into a debate. Someone swaps the embedding model and generation quality drops; is that the embeddings, the chunking that changed at the same time, the prompt someone edited on Tuesday, or a provider model update nobody was told about? With per-stage metrics the question is answerable in ten minutes.

Second, it gives you a ceiling. If recall@10 is 0.6, no prompt engineering takes end-to-end accuracy above 0.6, because in 40 percent of cases the evidence is not in the context. Knowing the ceiling stops teams from spending a quarter optimising the half that is already working.

Mechanics

The golden set

Everything depends on a labelled evaluation set, and it is the artifact people skip because it is unglamorous.

Size: 100 to 500 question-and-source pairs. Below 100 the confidence intervals swallow the effect you are measuring; above 500 you are spending labelling budget that would be better spent on coverage.

Composition, and this is the part that matters. Do not sample uniformly from query logs, because that gives you 80 percent head queries and you will optimise for questions that already work. Stratify:

Stratum	Share	Why
Head queries (most frequent)	30%	Protects the common case from regression
Tail queries (rare, specific)	30%	Where retrieval actually fails
Known-failure queries	20%	Harvested from thumbs-down and support tickets
Adversarial and out-of-scope	20%	Should the system abstain? Does it?

That last stratum is the one nobody builds and it is where the reputational risk lives. A system that confidently answers a question it should have declined is worse than one that fails visibly.

Label the source, not the answer. Labelling "which document contains this" is fast and objective. Labelling "what the ideal answer is" is slow, subjective, and goes stale the moment the corpus changes. Source labels give you retrieval metrics immediately, and generation metrics can be computed against the retrieved context without a reference answer.

Retrieval metrics, and which to use

def recall_at_k(retrieved_ids, relevant_ids, k):
    """Did we get the evidence into the context at all? This is the ceiling
    on end-to-end quality, so it is the metric to optimise first."""
    return len(set(retrieved_ids[:k]) & set(relevant_ids)) / len(relevant_ids)

def mrr(retrieved_ids, relevant_ids):
    """Reciprocal of the rank of the first relevant document. Right metric
    when there is one correct answer and position matters."""
    for rank, doc in enumerate(retrieved_ids, 1):
        if doc in relevant_ids:
            return 1.0 / rank
    return 0.0

def ndcg_at_k(retrieved_ids, relevance, k):
    """Graded relevance with position discount. Use when documents are
    partially relevant rather than binary, e.g. a search results page."""
    import math
    dcg = sum(relevance.get(d, 0) / math.log2(i + 1)
              for i, d in enumerate(retrieved_ids[:k], start=1))
    ideal = sorted(relevance.values(), reverse=True)[:k]
    idcg = sum(r / math.log2(i + 1) for i, r in enumerate(ideal, start=1))
    return dcg / idcg if idcg else 0.0

For RAG specifically, recall@k is the metric that matters most, because the generator will read all $k$ chunks regardless of order. NDCG matters when $k$ is large enough that position affects what the model attends to, which given the position effect it does past a handful of chunks.

Generation metrics

Groundedness / faithfulness. Decompose the answer into atomic claims, and check each against the retrieved context. This is the metric that catches confabulation, and it is computable without a reference answer, which is why it is the most practical generation metric.

FAITHFULNESS_PROMPT = """Given the CONTEXT and a CLAIM, answer with exactly one word.

Answer SUPPORTED if the claim follows from the context.
Answer UNSUPPORTED if it does not, including if it is true in general but
not stated in the context.

CONTEXT:
{context}

CLAIM:
{claim}"""

def faithfulness(answer, context, judge):
    claims = extract_claims(answer, judge)          # one LLM call
    verdicts = [judge(FAITHFULNESS_PROMPT.format(context=context, claim=c))
                for c in claims]                    # one call per claim
    return sum(v.strip().upper() == "SUPPORTED" for v in verdicts) / len(claims)

Note the instruction "including if it is true in general but not stated". Without it, a judge marks generally-true claims as supported, and you stop measuring grounding at all.

Answer relevance. Does the answer address the question asked? Cheapest robust implementation: have a model generate questions the answer would answer, embed them, and measure similarity to the original question. Catches the failure where the model answers a related but different question.

Context precision. What fraction of the retrieved chunks were actually used? Low precision with high recall means you are paying for distractors, which both costs tokens and hurts quality through the position effect.

LLM-as-judge, and its three biases

Human labelling does not scale to every CI run, so the judge is usually a model. Zheng et al. documented its failure modes, and naming them is what separates a credible evaluation story from a naive one:

Position bias. Given two answers to compare, judges systematically favour one position. Mitigation: evaluate both orderings and require consistency; count disagreements as ties.

Verbosity bias. Longer answers score higher independent of quality. Mitigation: instruct explicitly that length is not quality, and monitor the correlation between answer length and score as a diagnostic. If it is strongly positive, your judge is measuring length.

Self-preference bias. A model favours text produced by itself or its own family. Mitigation: use a different model family as judge than as generator.

Two further practices that materially improve judge reliability: rubric-based scoring with explicit criteria rather than a 1-to-10 vibe, and reference-guided grading where a reference answer is available. And critically: validate the judge against human labels on a sample. A judge whose agreement with humans is 0.6 is not a measurement instrument, and you cannot know that without checking.

Gating CI

# Runs on any change to prompts, models, chunking, or retrieval config.
# Absolute floors catch drift; deltas catch regressions the floors miss.
evaluation_gate:
  retrieval:
    recall_at_10:        { min: 0.85, max_delta: -0.02 }
    context_precision:   { min: 0.60 }
  generation:
    faithfulness:        { min: 0.90, max_delta: -0.03 }
    answer_relevance:    { min: 0.85 }
  abstention:
    out_of_scope_refusal_rate: { min: 0.90 }   # must decline what it cannot answer
  cost:
    tokens_per_query_p95: { max: 6000 }        # a quality gain that triples cost is a trade

Two things this encodes that teams usually miss. Abstention is a gated metric, so a change that improves answers by making the system answer everything fails. And cost is a gate, because otherwise the eval suite rewards throwing more context at the problem.

The other essential discipline: pin model versions. A provider updating a model underneath you is a silent behaviour change, and without pinning you cannot tell it from your own regression.

A worked example

A support assistant. Reported problem: "answers are getting worse". No one can say how.

Week 1, build the golden set. 200 questions stratified as above; support engineers label the source document, taking about six hours in total. Baseline:

Metric	Value
recall@10	0.79
context precision	0.31
faithfulness	0.94
answer relevance	0.88
out-of-scope refusal	0.42

The diagnosis takes about an hour. Faithfulness is high, so the model is not confabulating; it is using the context it gets. Recall of 0.79 is the ceiling, so 21 percent of questions are unanswerable by construction. Context precision of 0.31 means roughly two thirds of retrieved chunks are noise. And refusal at 0.42 is the actual reported problem: the system answers well over half the questions it should decline, which is what users experience as "getting worse" even though nothing regressed.

Three fixes, in order of measured return:

Abstention gate. Refuse when the top reranker score is below a threshold tuned on the golden set. Refusal rate 0.42 to 0.91, at the cost of declining 3 percent of answerable questions. Largest user-visible improvement, and it is configuration, not modelling.
Reduce $k$ from 12 to 5 after adding a reranker. Context precision 0.31 to 0.68, faithfulness 0.94 to 0.96 (fewer distractors), tokens per query down 55 percent. A quality and cost win, which is unusual and comes from stopping doing something.
Hybrid retrieval to attack the recall ceiling. recall@10 0.79 to 0.88.

Then the regression that the suite caught two months later. A prompt change to make answers more concise dropped faithfulness from 0.96 to 0.89, because the model compressed by dropping qualifiers that were doing the grounding work. Blocked in CI. Without the split metrics, that ships and shows up as a support escalation about a confidently wrong answer six weeks later.

Production evidence

Ragas is the most widely used open-source RAG evaluation framework and implements exactly this split: faithfulness and answer relevancy on the generation side, context precision and context recall on the retrieval side. Its existence and adoption is good evidence the two-family split is standard practice rather than a personal framework.

TruLens implements the same decomposition as the "RAG triad" (context relevance, groundedness, answer relevance), independently arriving at the same structure.

Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023) is the primary source for position bias, verbosity bias and self-enhancement bias in model judges, and for the mitigations (swapping positions, few-shot examples, reference-guided grading).

Anthropic's contextual retrieval work reported its results as retrieval failure rate at fixed $k$ rather than as end-to-end accuracy, which is the same discipline: measure the stage you changed.

The debate

The alternative is end-to-end evaluation only: score final answers against human judgement or a reference, and treat the system as a black box. It has a real advantage, which is that it measures what users experience, and no stage-wise metric guarantees the whole works.

Its weakness is that it cannot direct work. A drop in end-to-end accuracy tells you something is wrong and nothing about where, and in a system with five moving parts that is an expensive investigation every time.

The other alternative is online metrics only: thumbs, escalation rate, abandonment. These are the ground truth and they are also slow, noisy, confounded by everything else shipping that week, and unavailable before launch.

My position: stage-wise offline metrics to direct engineering, a small end-to-end set to catch what the stages miss, and online metrics as the arbiter of whether any of it mattered. Gate CI on the offline suite, because that is the only one fast enough. And track offline-to-online correlation as a metric in its own right: when your suite stops predicting production outcomes, fix the suite before the system.

Stage-wise evaluation is the wrong emphasis when the system is a prototype with no users, where the honest answer is to ship it to five people and read the transcripts; and when the retrieval stage is trivially correct (a small, well-structured corpus), in which case generation metrics alone are enough.

Follow-up Q&A

"Why separate retrieval evaluation from generation evaluation?" Because they have different failure modes, different fixes and different owners, and a combined number cannot tell you which one broke. Retrieval sets the ceiling: if recall@10 is 0.6, no prompt work takes you above 0.6. Generation tells you whether the model used what it was given. The context injection test is the manual version of the same split: paste the correct passage into the context and see whether the answer becomes right.

"How do you build a golden set without labelling budget?" Harvest it. Take the 100 most frequent queries and the 50 that produced thumbs-down, and label the source document rather than the ideal answer, which is fast and objective. That gives you retrieval metrics immediately, and generation metrics can be computed against the retrieved context with no reference answer at all. Stratify so you are not measuring only head queries, and include an out-of-scope stratum, because abstention behaviour is where the reputational risk is.

"Your LLM judge says quality is fine and users disagree. What now?" Validate the judge, which most teams never do. Take 50 examples, have humans label them, and measure agreement. If agreement is poor the judge is the problem: check for verbosity bias by correlating score with answer length, check position bias by running comparisons in both orders, and check whether the rubric is specific enough to be applied consistently. If agreement is good but users still disagree, your metric is measuring the wrong thing, and the fix is to look at what users actually complained about and add a metric for it.

"What do you gate in CI, and at what threshold?" Absolute floors plus maximum deltas on recall@k, faithfulness and answer relevance, so you catch both drift and regressions. Plus two that get forgotten: an abstention floor, so a change cannot improve answers by answering everything; and a cost ceiling, so it cannot improve quality by tripling context. Thresholds come from the current baseline minus a tolerance sized to the suite's noise, which you measure by running the suite three times on identical inputs.

"How do you handle model version drift?" Pin the version explicitly, never use a floating alias in production, and re-run the full suite on every provider update before adopting it. Without pinning, a provider change is indistinguishable from your own regression, and you will spend days bisecting your own commits. Also record the model version in every trace, so a production incident can be correlated with a version change after the fact.

Common misconceptions

The most common is that one accuracy number is enough. It cannot direct work, which is the only thing a metric is for.

The second is that a reranker improves recall. It reorders the retrieved set; recall@k for the same $k$ is unchanged by construction. Reranking improves precision metrics, and confusing the two produces months of work on the wrong stage.

The third is that an LLM judge is a measurement instrument out of the box. It is a model with documented biases, and until you have validated it against human labels you do not know what it measures.

Interview delivery note

Say this: "Two metric families, never one. Retrieval: recall@k, and it's the ceiling, because if the evidence isn't in the context no prompt work fixes it. Generation: faithfulness, meaning every claim traceable to the retrieved context, and answer relevance. A single accuracy number can't tell me whether to fix the retriever or the prompt, so it can't direct any work."

Then the two things that make it credible: "I'd build a golden set of 100 to 500 examples, stratified so it isn't all head queries and including an out-of-scope stratum, and I'd label the source document rather than the ideal answer because that's fast and objective. And I'd validate the judge against human labels, because an LLM judge has documented position, verbosity and self-preference biases and until you've measured agreement you don't know what it's measuring."

The depth signal is gating abstention and cost in CI alongside quality, because that shows you have seen the failure where a change improves answers by answering everything, or by tripling the context.

The Staff & Team Lead Interview Reference