Evaluating a RAG system requires measuring retrieval quality, answer faithfulness, and system performance separately. High recall with low faithfulness means your system retrieves correct documents but hallucinates the answer. No single metric captures the full picture — you need a scorecard, not a single number.
You deploy a RAG system. Users type questions. Answers appear. The answers look reasonable.
Is the system working well?
You cannot answer that question by reading individual answers. A system can retrieve the wrong documents and still produce a fluent, plausible response. A system can retrieve perfect documents and generate a hallucinated answer. The only way to know whether a RAG system works is to measure it systematically.
This tutorial teaches you how — from the metrics that matter to a Python benchmark you can run today.
This tutorial builds on the RAG Explained and RAG Architecture articles. If you haven't read those, start there for the foundational concepts.
1. Why "Looks Good" Is Never Enough
When a human reads a RAG answer and thinks "that seems right," they're performing a very imprecise evaluation. Here's why that's dangerous:
| What You See | What Might Actually Be Happening |
|---|---|
| Fluent, confident answer | Hallucinated from the model's training data, not your documents |
| Answer mentions correct topic | Retrieved wrong documents, but the topic happened to overlap |
| Answer cites a source | Citation doesn't actually support the claim in the answer |
| Answer is fast | Chunks are too small, context is too narrow, key information was missed |
| Most answers seem correct | The 20% that are wrong are the ones users remember and complain about |
Systematic evaluation catches these problems before users encounter them.
2. The Three Pillars of RAG Evaluation
RAG evaluation has three independent dimensions. Measuring only one or two will miss critical failures.
Did we find the right documents?
Precision@K, Recall@K, MRR, nDCG
Is the answer faithful and relevant?
Faithfulness, Answer Relevance, Citation Accuracy
Does it perform well enough to use?
Latency, Cost, Throughput, Error Rate
A system that retrieves perfect documents but generates hallucinated answers is broken. A system with accurate answers that takes 30 seconds per query is unusable in production. You need all three.
3. Retrieval Metrics Explained
Precision@K
Of the K documents retrieved, how many are actually relevant?
Example: retrieved 3 docs, 2 are relevant → Precision@3 = 0.67
When it matters: When context-window space is limited and every chunk costs tokens. High precision means you're not wasting tokens on irrelevant documents.
Recall@K
Of all the relevant documents in your corpus, how many did you find?
Example: 5 relevant docs exist, top 3 retrieved 2 → Recall@3 = 0.40
When it matters: When completeness matters. A medical RAG system that misses 60% of relevant documents could give dangerously incomplete answers.
MRR (Mean Reciprocal Rank)
How high up in the results does the first relevant document appear?
First relevant at position 3 → score = 1/3 = 0.33
First relevant at position 5 → score = 1/5 = 0.20
When it matters: When users only read the top result. MRR tells you how often they find what they need immediately.
Combining Retrieval Metrics
No single retrieval metric tells the full story:
| Scenario | P@3 | R@3 | MRR | Diagnosis |
|---|---|---|---|---|
| All 3 retrieved are relevant | 1.00 | 1.00 | 1.00 | Perfect |
| 1 relevant in top 3, but 5 exist | 0.33 | 0.20 | 0.33 | Poor recall |
| 2 relevant in top 3, but ranked low | 0.67 | 0.67 | 0.50 | Ranking issue |
4. Answer Quality Metrics Explained
Retrieval metrics tell you whether the right documents were found. Answer quality metrics tell you whether the LLM used those documents correctly.
Faithfulness
Does the answer actually come from the retrieved documents?
A faithful answer only makes claims that are supported by the retrieved chunks. An unfaithful answer contains information the model invented — even if that information happens to be true from the model's training data.
Answer Relevance
Does the answer actually address the question?
A relevant answer uses keywords and concepts from the expected answer. A non-relevant answer might be about the right topic but miss the specific question asked.
Citation Accuracy
Do the cited sources actually support the claims?
The system cites document X. Does document X actually contain the information the answer claims? Citation accuracy measures this alignment.
5. System Performance Metrics
Even a perfectly accurate RAG system is useless if it's too slow or too expensive.
| Metric | What It Measures | Target (Interactive) |
|---|---|---|
| End-to-end latency | Query → final answer | < 2 seconds |
| Retrieval latency | Query → documents returned | < 200ms |
| Generation latency | Context assembled → answer complete | < 1.5 seconds |
| Cost per query | Embedding + LLM tokens | < $0.01 |
| Throughput | Queries per second | Depends on SLA |
6. Building a Ground Truth Dataset
Every metric above requires a ground truth — a set of questions with known correct answers and known relevant documents. Without ground truth, you cannot compute precision, recall, faithfulness, or citation accuracy.
What a Ground Truth Entry Looks Like
"question": "What is the Python GIL?",
"expected_answer": "The Global Interpreter Lock is a mutex that prevents...",
"relevant_docs": ["python_gil.txt", "threading_basics.txt"]
}
Creating Ground Truth
| Approach | Quality | Scalability | Cost |
|---|---|---|---|
| Expert-created Q&A pairs | Highest | Low | High (time) |
| User logs filtered and verified | High | Medium | Medium |
| LLM-generated Q&A from documents | Medium (needs review) | High | Low |
| Synthetic from document structure | Low-Medium | Very high | Very low |
A minimum viable ground truth dataset for a production RAG system should contain at least 50–100 question-answer pairs spanning your key topic areas.
7. A Python RAG Evaluation Benchmark
The following benchmark uses zero external dependencies. It demonstrates how different RAG systems score across the same metrics using a synthetic document corpus.
# Measures: Precision@K, Recall@K, MRR, Faithfulness, Citation Accuracy
import re, math, time
# ─── Ground Truth ────────────────────────────────────────────
GROUND_TRUTH = [
{"question": "What is the Python GIL?",
"expected": "Global Interpreter Lock mutex thread",
"relevant_docs": ["gil.txt", "threading.txt"]},
{"question": "How do you create a virtual environment?",
"expected": "python -m venv activate pip",
"relevant_docs": ["venv.txt", "pip.txt"]}
]
# ─── Retrieval Metric: Precision@K ──────────────────────────
def precision_at_k(retrieved, relevant, k=3):
# Count how many of the top-K are relevant
top_k = retrieved[:k]
relevant_set = set(relevant)
hits = sum(1 for d in top_k if d in relevant_set)
return hits / k
# ─── Retrieval Metric: Recall@K ─────────────────────────────
def recall_at_k(retrieved, relevant, k=3):
top_k = retrieved[:k]
relevant_set = set(relevant)
hits = sum(1 for d in top_k if d in relevant_set)
return hits / len(relevant) if relevant else 0
# ─── Retrieval Metric: MRR ──────────────────────────────────
def mrr(retrieved, relevant):
for i, doc in enumerate(retrieved):
if doc in relevant:
return 1 / (i + 1)
return 0
# ─── Generation Metric: Faithfulness ────────────────────────
def faithfulness(answer, docs):
# Fraction of answer keywords found in retrieved docs
ans_words = set(re.findall(r'[a-z]+', answer.lower()))
doc_text = ' '.join(docs).lower()
doc_words = set(re.findall(r'[a-z]+', doc_text))
if not ans_words: return 0
return len(ans_words & doc_words) / len(ans_words)
# ─── Generation Metric: Citation Accuracy ───────────────────
def citation_accuracy(citations, relevant):
if not citations: return 0
correct = sum(1 for c in citations if c in relevant)
return correct / len(citations)
Save this as rag_eval.py and run it with python rag_eval.py. The benchmark compares four simulated RAG systems (good, mediocre, poor, hallucinating) on the same 8-question ground truth dataset.
8. Benchmark Results: Why Metrics Differ
The benchmark reveals a critical finding — systems with identical retrieval scores can have completely different answer quality:
| System | P@3 | R@3 | MRR | Faithful | CitAcc | AnsRel |
|---|---|---|---|---|---|---|
| Good RAG | 0.17 | 0.44 | 0.39 | 1.00 | 0.19 | 0.05 |
| Mediocre RAG | 0.17 | 0.44 | 0.39 | 0.00 | 0.00 | 0.05 |
| Hallucinating RAG | 0.17 | 0.44 | 0.39 | 0.00 | 0.00 | 0.00 |
9. The Evaluation Workflow
Follow this workflow to evaluate any RAG system:
- Build Ground Truth — 50+ question-answer pairs with relevant documents
- Run Retrieval — For each question, retrieve top-K documents
- Measure Retrieval — Compute Precision@K, Recall@K, MRR
- Generate Answers — Send retrieved context + question to LLM
- Measure Faithfulness — Does the answer come from the retrieved documents?
- Measure Answer Relevance — Does the answer address the question?
- Measure Citation Accuracy — Do cited sources support the claims?
- Measure System Performance — Latency, cost, throughput
- Aggregate and Compare — Build a scorecard, track over time
- Identify Weaknesses — Which questions fail? Why? Fix and retest.
10. Common Evaluation Mistakes
| Mistake | Why It's Wrong | What To Do Instead |
|---|---|---|
| Only measuring retrieval | Misses hallucination and generation errors | Add faithfulness and citation accuracy |
| Evaluating on 5 questions | Too small for statistical confidence | Use 50–100+ questions minimum |
| Only using easy questions | Doesn't test edge cases or complex queries | Include ambiguous, multi-part, and edge-case questions |
| Evaluating once and never re-evaluating | Documents change, models update, quality drifts | Set up automated evaluation in CI/CD |
| Trusting only automated metrics | Automated metrics can miss subtle quality issues | Combine automated metrics with human review |
11. Evaluation Scorecard Template
Use this scorecard to evaluate any RAG system. Clearly label it as an internal evaluation framework, not a certification:
| Metric | 0 (Poor) | 1-2 (Weak) | 3 (Adequate) | 4-5 (Strong) |
|---|---|---|---|---|
| Precision@3 | < 0.1 | 0.1–0.3 | 0.3–0.6 | > 0.6 |
| Recall@3 | < 0.2 | 0.2–0.5 | 0.5–0.8 | > 0.8 |
| MRR | < 0.1 | 0.1–0.3 | 0.3–0.6 | > 0.6 |
| Faithfulness | < 0.2 | 0.2–0.5 | 0.5–0.8 | > 0.8 |
| Citation Accuracy | 0 | 0.0–0.2 | 0.2–0.5 | > 0.5 |
| Latency (p95) | > 5s | 3–5s | 1–3s | < 1s |
12. When to Evaluate
RAG evaluation is not a one-time activity. Evaluate at each of these stages:
| Stage | What to Evaluate | Frequency |
|---|---|---|
| Before deployment | Full benchmark against ground truth | Once (minimum) |
| After model update | Regression test against baseline | Each update |
| After document update | Retrieval quality, citation accuracy | Each major update |
| In production | Latency, error rate, user feedback | Continuous |
| Periodic review | Full benchmark, human review sample | Monthly/quarterly |
13. Practical Exercises
14. FAQ
15. Conclusion
Evaluating a RAG system requires measuring three independent dimensions: retrieval quality, answer faithfulness, and system performance. Our benchmark demonstrated that systems with identical retrieval scores can have completely different answer quality — and that faithfulness and citation accuracy are the metrics that catch hallucination.
The most dangerous RAG failure is the one that looks correct: a fluent answer that happens to be grounded in the wrong documents, or not grounded in any documents at all. The only way to detect this systematically is with a comprehensive evaluation scorecard.
Start with Recall@K and Faithfulness. Build your ground truth dataset. Run the benchmark. Then iterate.
For the foundational concepts behind RAG, see RAG Explained. For the full system architecture, see RAG Architecture Explained. For a practical production guide, see From RAG Prototype to Production.
Related BestWordz Tools and Articles
Complete beginner-to-advanced guide RAG Architecture Explained
Every component of a RAG system Why RAG Systems Still Hallucinate
Common failure modes and mitigations Reranking in RAG
Why vector search alone isn't enough From RAG Prototype to Production
Building reliable AI knowledge systems Embeddings Explained
How AI converts meaning into numbers Regex Tester
Test the regex patterns used in evaluation