Why RAG Exists: The Hallucination Problem

Retrieval-Augmented Generation (RAG) is the most practical technique for making LLMs answer questions using your own documents. This tutorial explains every component — from raw text to grounded, cited answers — with a complete Python project you can run today.

Why RAG Exists: The Hallucination Problem

Ask an LLM: "What is our company's refund policy?"

Without RAG, the model might:

• Generate a plausible but incorrect policy
• Use training data from other companies
• Give a generic answer that doesn't match your actual documents

This isn't a bug — it's how LLMs work. They predict likely text based on patterns in training data. They don't "look up" your documents.

RAG solves this by searching your documents first, then giving the LLM only the relevant pieces as context.

The RAG Pipeline: Every Component Explained

RAG has seven core components. Weakness in any one degrades the entire system.

1. Documents (Ingestion)

RAG starts with your data. This can be:

• PDFs — policies, manuals, research papers
• Web pages — documentation, articles, FAQs
• Markdown/Text — notes, READMEs, specs
• Databases — structured records exported as text
• Emails/Chat — support conversations, threads

The ingestion step extracts clean text from these sources. PDF parsing, HTML stripping, and encoding normalization happen here.

2. Chunking

LLMs have limited context windows. You can't send entire libraries. Chunking splits documents into smaller, overlapping pieces.

# Word-level chunking with overlap
def chunk_text(text, chunk_size=100, overlap=20):
    words = text.split()
    chunks, start = [], 0
    while start < len(words):
        end = start + chunk_size
        chunks.append(" ".join(words[start:end]))
        start += chunk_size - overlap  # overlap preserves context
    return chunks

Why overlap? Without it, a sentence split across two chunks loses its meaning. Overlap ensures each chunk has enough surrounding context to be understood independently.

3. Embeddings

Each chunk needs to be converted into a numerical vector that captures its meaning. This is what embeddings do.

A good embedding model places semantically similar texts close together in vector space. "Python functions" and "defining functions" end up near each other. "Python functions" and "car engines" end up far apart.

Popular embedding models:

4. Vector Database

The vector database stores all chunk embeddings and enables fast similarity search. When a query arrives, it finds the most similar chunks.

Common options:

• FAISS — Facebook's library, fast, local, no server needed
• Chroma — lightweight, developer-friendly, good for prototyping
• Qdrant — production-ready, filtering, scaling
• Pinecone — fully managed cloud service
• Weaviate — hybrid search, GraphQL API

5. Retrieval

When a user asks a question, the system:

1. Converts the question into an embedding
2. Searches the vector database for the top-k most similar chunks
3. Returns the chunks with similarity scores

Top-k is typically 3–10 chunks. Too few → missing relevant information. Too many → diluted context and higher cost.

6. Reranking

Initial retrieval is fast but approximate. Reranking re-scores the top results using a more precise model.

Learn more in our Reranking in RAG tutorial.

7. Context Assembly and Generation

The final step assembles retrieved chunks into a context string and sends it to the LLM with the user's question.

# Context assembly
context = ""
for i, (chunk, score) in enumerate(retrieved_chunks):
    context += f"[Source {i+1}: {chunk.title}]\n{chunk.text}\n\n"

# LLM prompt with context
prompt = f"""Answer the question using ONLY the provided context.

Context:
{context}

Question: {question}

Cite sources [Source N] in your answer."""

Complete Python Project: Mini-RAG System

Here's a working RAG system you can run with zero dependencies. It demonstrates every step of the pipeline using TF-IDF vectors instead of neural embeddings.

import math, re
from collections import Counter

# 1. CHUNKING
def chunk_text(text, size=100, overlap=20):
    words = text.split()
    chunks, start = [], 0
    while start < len(words):
        chunks.append(" ".join(words[start:start+size]))
        start += size - overlap
    return chunks

# 2. TF-IDF EMBEDDINGS
def build_idf(docs):
    n = len(docs)
    df = Counter()
    for doc in docs:
        for t in set(re.findall(r'[a-z]+', doc.lower())):
            df[t] += 1
    return {t: math.log((n+1)/(c+1))+1 for t,c in df.items()}

def tfidf(text, idf):
    tokens = re.findall(r'[a-z]+', text.lower())
    tf = Counter(tokens)
    total = len(tokens) or 1
    return {t: (c/total)*idf.get(t,1) for t,c in tf.items()}

def cosine(a, b):
    common = set(a) & set(b)
    dot = sum(a[k]*b[k] for k in common)
    return dot / (math.sqrt(sum(v**2 for v in a.values()))
              * math.sqrt(sum(v**2 for v in b.values())))

# 3. RETRIEVAL
idf = build_idf(chunks)
q_vec = tfidf("What web frameworks does Python have?", idf)
scores = [(i, cosine(q_vec, tfidf(c, idf))) for i,c in enumerate(chunks)]
top_3 = sorted(scores, key=lambda x: x[1], reverse=True)[:3]

# 4. CONTEXT + ANSWER
context = "\n\n".join(chunks[i] for i,_ in top_3)
# Send context + question to your LLM

The full demo with evaluation, reranking, and mock LLM is available in the article repository. It runs with zero external dependencies.

Evaluation: How Do You Know It Works?

RAG evaluation measures two things: retrieval quality and answer quality.

Retrieval Metrics

Answer Quality Metrics

• Faithfulness — Is the answer supported by the retrieved context?
• Answer Relevance — Does the answer actually address the question?
• Context Relevance — Is the retrieved context actually useful?

See our RAG Evaluation tutorial for detailed benchmarks.

Common RAG Mistakes

Read more in Why RAG Systems Still Hallucinate.

Production Considerations

Building a prototype is easy. Making it production-ready requires addressing:

• Document updates — How do you re-index when documents change?
• Access control — Different users should see different documents
• Latency — Vector search + reranking + LLM = multiple network calls
• Cost — Embedding API calls, LLM tokens, vector DB hosting
• Monitoring — Track retrieval quality, answer quality, and user satisfaction
• Failure recovery — What happens when the vector DB is unavailable?

See our From RAG Prototype to Production guide.

Related BestWordz Tutorials

• Embeddings Explained: How AI Converts Meaning into Numbers
• RAG Architecture Explained: Every Component
• Reranking in RAG: Why Vector Search Alone Is Not Enough
• Why RAG Systems Still Hallucinate
• RAG Evaluation: How to Measure Retrieval and Answer Quality
• From RAG Prototype to Production
• Hybrid Search: Combining BM25 and Vector Search
• MCP + RAG: Connecting AI Agents to Private Knowledge
• RAG Security: Protecting Vector Stores
• Context Engineering Explained

FAQ

Discuss this topic on BestWordz Community.