🎯 Key Takeaway: RAG doesn't eliminate hallucination — it moves the problem from the model to the retrieval layer. Understanding the 5 root causes helps you build systems that hallucinate less.
You built a RAG system. You added a vector store. You connected an LLM. You expected grounded, accurate answers.
But the system still hallucinates.
Why?
Because RAG doesn't solve hallucination. It moves the problem from "the model doesn't know" to "the system retrieved the wrong information" — or worse, "the system retrieved the right information but the model ignored it."
This article explains the 5 root causes of RAG hallucination and how to fix each one.
The 5 Root Causes
| # | Cause | Problem | Mitigation |
|---|---|---|---|
| 1 | Bad Retrieval | Irrelevant chunks retrieved | Better embeddings, reranking |
| 2 | Wrong Chunks | Chunking destroys context | Semantic chunking, overlap |
| 3 | Incomplete Context | Missing information | Increase K, multi-query |
| 4 | Model Behavior | LLM ignores context | Low temperature, better prompts |
| 5 | Conflicting Documents | Multiple sources disagree | Deduplication, source priority |
1. Bad Retrieval: The Wrong Chunks Come Back
The most common cause: the vector store returns chunks that are semantically similar but not relevant.
Example
Query: "What is the refund policy for international orders?"
Retrieved chunks:
1. "Our general return policy allows..." (similarity: 0.82) ← General
2. "International shipping rates vary..." (similarity: 0.79) ← Wrong topic
3. "Refund requests must be submitted within 30 days..." (similarity: 0.76) ← This is the answer!
The answer is ranked #3, not #1.Why It Happens
- Embedding quality: Bi-encoders compress meaning into fixed vectors, losing nuance
- Vocabulary mismatch: "refund" and "return" are similar but different
- No interaction: Query and document encoded independently
Mitigations
| Strategy | How It Helps | Impact |
|---|---|---|
| Hybrid Search | Combine semantic + keyword | +15-25% precision |
| Reranking | Cross-encoder rescores results | +20-40% precision |
| Query Expansion | Generate multiple query variants | +10-20% recall |
| Metadata Filtering | Filter by source, date, type | Reduces noise |
# Hybrid search example
results = vector_store.search(
query,
search_type="hybrid", # semantic + keyword
k=20
)
# Rerank results
reranked = cross_encoder.rerank(query, results, top_k=5)2. Wrong Chunks: Chunking Destroys Context
Documents are split into chunks before embedding. If the answer spans multiple chunks, the system may retrieve only part of it.
Example
Original document:
"The refund policy requires a receipt. Returns must be made within 30 days.
International orders have a 60-day window. Shipping costs are non-refundable."
After chunking:
Chunk 1: "The refund policy requires a receipt. Returns must be made within 30 days."
Chunk 2: "International orders have a 60-day window. Shipping costs are non-refundable."
Query: "What is the international refund policy?"
→ Retrieves Chunk 2 only (missing the receipt requirement)Why It Happens
- Fixed-size chunking: Splits at arbitrary boundaries
- No overlap: Context lost at chunk edges
- Wrong granularity: Chunks too small or too large
Mitigations
| Strategy | How It Helps |
|---|---|
| Semantic Chunking | Split at meaning boundaries |
| Overlap | 10-20% overlap preserves context |
| Parent-Child | Retrieve child, return parent |
| Sentence-Level | Split at sentence boundaries |
# Semantic chunking with overlap
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200, # Preserves context at boundaries
separators=["\n\n", "\n", ". ", " ", ""]
)3. Incomplete Context: Missing Information
Sometimes the answer exists in the knowledge base, but the retrieval doesn't find it — or doesn't retrieve enough chunks.
Example
Query: "What are all the exceptions to the refund policy?"
Retrieved: 3 chunks
Actual answer: Spread across 5 chunks
Result: LLM generates partial answer, misses 2 exceptionsWhy It Happens
- K too small: Not enough chunks retrieved
- Distributed information: Answer spans multiple documents
- Single query: One query may not capture all aspects
Mitigations
# Multi-query retrieval
def multi_query_retrieval(query: str, k: int = 5):
"""Generate multiple queries and retrieve for each"""
# Generate query variants
queries = [
query,
f"What are the exceptions to {query}?",
f"List all details about {query}",
f"What are the conditions for {query}?"
]
# Retrieve for each query
all_results = []
for q in queries:
results = vector_store.search(q, k=k)
all_results.extend(results)
# Deduplicate and rerank
unique_results = deduplicate(all_results)
reranked = cross_encoder.rerank(query, unique_results, top_k=k)
return reranked4. Model Behavior: The LLM Ignores Context
Even with perfect retrieval, the LLM may ignore the context and generate based on its training data.
Example
Context provided:
"Our refund policy requires a receipt and must be processed within 30 days."
LLM response:
"According to our policy, refunds can be processed within 60 days without a receipt."
← LLM hallucinated despite having the correct context!Why It Happens
- High temperature: Increases creativity, decreases grounding
- Poor prompting: LLM not instructed to use context
- Context length: LLM may "lose" information in long contexts
- Training bias: LLM prefers its own knowledge
Mitigations
| Strategy | How It Helps |
|---|---|
| Low Temperature | 0.0-0.3 for grounded answers |
| Better Prompts | "Use ONLY the provided context" |
| Citation Instructions | "Cite the source for each claim" |
| Output Validation | Verify answer against sources |
# Better prompt template
prompt = """You are a helpful assistant. Answer the question using ONLY
the provided context. If the context doesn't contain the answer,
say "I don't have enough information."
For each claim, cite the source using [Source X] format.
Context:
{context}
Question: {question}
Answer (with citations):"""
# Generation with low temperature
response = llm.generate(
prompt,
temperature=0.1, # Low = more grounded
top_p=0.9
)5. Conflicting Documents: Multiple Sources Disagree
Your knowledge base may contain contradictory information from different sources or time periods.
Example
Retrieved chunks:
1. "Refund policy: 30 days" (from policy_v1.pdf, 2024)
2. "Refund policy: 60 days" (from policy_v2.pdf, 2025)
3. "Refund policy: 45 days" (from faq.md, 2025)
LLM: "The refund policy is... uh... 30-60 days?" ← Confused by conflictWhy It Happens
- Outdated documents: Old versions still indexed
- Multiple sources: Different departments, different policies
- No priority: System doesn't know which source to trust
Mitigations
# Source priority and freshness
def prioritized_retrieval(query: str):
results = vector_store.search(query, k=10)
# Add metadata for filtering
for result in results:
result['freshness'] = calculate_freshness(result['metadata']['date'])
result['priority'] = get_source_priority(result['metadata']['source'])
# Sort by priority + freshness
results.sort(
key=lambda x: (x['priority'], x['freshness']),
reverse=True
)
# Return top-k from highest priority source
return results[:5]
⚠️ Important: Deduplication and conflict resolution require domain knowledge. An automated system may not know which source is authoritative.
Bonus: Prompt Engineering for Hallucination Reduction
The prompt can significantly reduce hallucination:
# Bad prompt (prone to hallucination)
prompt = f"""Context: {context}
Question: {question}
Answer:"""
# Better prompt (grounded)
prompt = f"""Answer the question using ONLY the provided context.
Rules:
1. If the context doesn't contain the answer, say "I don't have enough information"
2. Cite the source for each claim using [Source X]
3. Do not use any external knowledge
4. If the context is ambiguous, state what you found and what's missing
Context:
{context}
Question: {question}
Answer with citations:"""How to Measure Hallucination
| Metric | What It Measures | Tool |
|---|---|---|
| Faithfulness | Is the answer grounded in context? | RAGAS |
| Answer Relevancy | Does the answer address the question? | RAGAS |
| Context Precision | Are retrieved chunks relevant? | RAGAS |
| Context Recall | Is all needed information retrieved? | RAGAS |
| Citation Accuracy | Do citations match claims? | Custom |
# Using RAGAS for evaluation
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
# Evaluate your RAG system
result = evaluate(
dataset,
metrics=[faithfulness, answer_relevancy]
)
print(f"Faithfulness: {result['faithfulness']:.2f}")
print(f"Answer Relevancy: {result['answer_relevancy']:.2f}")Hallucination Reduction Checklist
| Check | Category | Status |
|---|---|---|
| Using hybrid search? | Retrieval | ☑️ |
| Added reranking? | Retrieval | ☑️ |
| Chunk overlap enabled? | Chunking | ☑️ |
| Retrieving enough chunks? | Context | ☑️ |
| Temperature set to 0.0-0.3? | Generation | ☑️ |
| Prompt instructs "use only context"? | Prompting | ☑️ |
| Citations requested? | Prompting | ☑️ |
| Output validated against sources? | Validation | ☑️ |
| Conflicts resolved? | Data | ☑️ |
| Using RAGAS or similar? | Evaluation | ☑️ |
Try It Yourself
Reduce hallucination in your RAG system with these BestWordz resources:
Conclusion
RAG doesn't eliminate hallucination. It moves the problem from the model to the system. The 5 root causes are:
- Bad Retrieval — Fix with hybrid search and reranking
- Wrong Chunks — Fix with semantic chunking and overlap
- Incomplete Context — Fix with multi-query retrieval
- Model Behavior — Fix with low temperature and better prompts
- Conflicting Documents — Fix with deduplication and source priority
The goal isn't zero hallucination — it's measurable, improvable grounding. Measure with RAGAS, iterate on each cause, and your system will hallucinate less over time.