Evaluate RAG retrieval before you judge the answers

A team builds a RAG assistant over its internal documents. The early demos go well. Then real users arrive, and the complaints start: answers that are vague, slightly wrong, or confidently made up. The usual next step is to rewrite the prompt, then try a larger model, then rewrite the prompt again. Some answers improve. Others get worse. Nobody can say whether the system is better overall.

That cycle has two root causes. First, the system is being judged by impressions instead of measurements. Second, it's being treated as one black box when it's really two components that fail in very different ways.

This post sets out a practical evaluation method we use for retrieval-augmented systems. It doesn't need special tooling or a large dataset. It needs a small set of good questions and the discipline to test the parts separately.

Two components, two kinds of failure

Every RAG system does two jobs:

Retrieval finds passages relevant to the question.
Generation writes an answer from those passages.

When the final answer is wrong, the cause is in one of these two places, and the fixes are completely different.

If the right passage was never retrieved, no prompt or model can produce a correct, grounded answer. The fix is in chunking, search, or ranking.
If the right passage was retrieved but the answer is still wrong, the problem is in generation: the prompt, the model, how much context it receives, or how the passages are presented to it.

Tuning prompts when retrieval is the problem is one of the most common ways teams lose weeks. The method below is designed to find out which problem you actually have before you spend effort fixing it.

Step 1: Build a small, honest question set

You don't need thousands of examples. Twenty to fifty well-chosen questions will show most of what you need to see, as long as they reflect what users really ask. For each question, write down:

The question, phrased as a real user would phrase it, not as an engineer would.
The source: the document and passage, or page, that contains the answer.
The expected answer, in a sentence or two.

The value of this set comes almost entirely from its difficulty. Deliberately include:

Exact-detail questions that depend on a specific number, date, name, or code.
Paraphrased questions that use different words from the documents.
Multi-part questions whose answer is spread across two or more passages.
Ambiguous questions with more than one reasonable reading.
Unanswerable questions whose answer isn't in the documents at all.

That last category is the one most often left out, and one of the most important. A system that never refuses to answer is one that makes things up.

The best source of questions is real usage: support tickets, search logs, or questions collected from the people who will use the system. If you have to write them yourself, have someone who didn't build the system do it.

Step 2: Measure retrieval on its own

Run each question through the retriever only, with no language model involved, and look at the ranked list of passages it returns. Two measures cover most of what you need.

Hit rate at k: the share of questions where the correct passage appears anywhere in the top k results. Pick k to match how many passages your system actually gives the model. If the model sees five passages, measure hit rate at 5.
Mean reciprocal rank (MRR): for each question, take one divided by the rank of the first correct passage (zero if it isn't found), then average across questions. This rewards systems that put the right evidence first, not just somewhere in the list.

Both are simple to compute:

def evaluate_retrieval(questions, retrieve, k=5):
    hits, reciprocal_ranks = 0, []
    for q in questions:
        results = retrieve(q["question"])[:k]
        rank = next((i for i, r in enumerate(results, 1) if r in q["relevant_ids"]), None)
        hits += rank is not None
        reciprocal_ranks.append(1 / rank if rank else 0)
    return {"hit_rate": hits / len(questions), "mrr": sum(reciprocal_ranks) / len(questions)}

Then read the numbers diagnostically:

Low hit rate means the right passages aren't being found at all. Look at chunking (is the answer split awkwardly across chunks?), at the search method (would keyword search catch exact terms that embeddings miss?), and at document quality (did tables or scanned pages extract cleanly?).
Reasonable hit rate but low MRR means the right passage is found but ranked too low. This is where a reranker usually earns its place.
Exclude unanswerable questions from these retrieval scores, since there is no correct passage to find. Score them separately in the next step.

Step 3: Judge the answers, with retrieval held fixed

Once retrieval is in reasonable shape, evaluate the generated answers. Score each answer on four separate questions instead of one overall impression:

Correct: does it match the expected answer?
Grounded: is every claim actually supported by the retrieved passages? An answer can be correct but ungrounded, if the model used its own background knowledge. That's risky, because next time the background knowledge may be wrong.
Properly cited: do the citations point to the passages that actually support each claim?
Honest about gaps: for unanswerable questions, does it say it doesn't know?

Keeping these separate matters. "Correct but ungrounded" and "grounded but wrong" are different problems with different fixes. A single overall score hides both.

Human review is the most reliable way to score these, and with a set of fifty questions it's very manageable. Language models can be used as judges to scale up, but only after you have checked that their scores agree with careful human scores on your own data.

Step 4: Read the failures

Metrics tell you how often the system fails. Only reading the failures tells you why. Take every failed question and put it into a category:

Not retrieved: the right passage wasn't in the results.
Ranked too low: it was found, but below the cutoff the model sees.
Retrieved but misused: the model had the evidence and still answered wrongly.
Made something up: the answer contains claims that aren't in any passage.
Should have refused: the question was unanswerable and the system answered anyway.
Bad question or label: the expected answer itself was wrong or ambiguous. This happens more often than people expect, and finding it improves the question set.

Count the categories. The largest one is where your next week of work should go. It's common to discover that a problem everyone blamed on "the model" is mostly a chunking problem, or that half the "hallucinations" were questions the documents never covered.

Step 5: Make it a habit

The first round of evaluation is useful. The real payoff comes from running it repeatedly.

Keep a baseline. Record the scores of the current system so every change has something to be measured against.
Re-run on every meaningful change: new chunking, a new embedding model, a reranker, a different prompt. A change that improves five questions and breaks eight others is now visible instead of hidden.
Grow the set from production. Every confusing real-world failure is a candidate for a new test question.
Watch for overfitting. If you tune heavily against the same fifty questions, keep a separate set aside that you only check occasionally.

The payoff

This method turns "the assistant feels unreliable" into specific, answerable questions: Is the evidence being found? Is it ranked well? Is the model using it faithfully? Does it know when to say "I don't know"? Each has a clear owner and a clear fix.

It also changes what conversations with stakeholders sound like. Instead of "we think it's better now," you can say "retrieval now finds the right passage for most questions where it previously missed, and the system refuses correctly on unanswerable questions." That's the difference between a promising demo and a system people can depend on.

For a worked example of the retrieval techniques mentioned here, see how we built hybrid retrieval for QuarterLens.