🎯 Key Takeaway: Vector search (bi-encoders) is fast but approximate. Reranking (cross-encoders) is slow but precise. The two-stage approach — retrieve with bi-encoder, rerank with cross-encoder — gives you the best of both worlds: speed and accuracy.
You ask: "What is the refund policy for international orders?"
Your vector search returns these results:
1. "Our general return policy allows..." (similarity: 0.82)
2. "International shipping rates vary..." (similarity: 0.79)
3. "Refund requests must be submitted within 30 days..." (similarity: 0.76) ← This is the answer!The perfect answer is ranked #3, not #1. Why?
Because vector search measures semantic similarity, not relevance. A document about "general returns" can have higher semantic similarity to "refund policy" than the actual refund policy document.
This is where reranking comes in.
The Problem with Vector Search Alone
Vector search uses bi-encoders to compress documents into fixed-size vectors. This compression loses information:
What Bi-Encoders Do
# Bi-encoder: Encode separately, compare vectors
query_vector = bi_encoder.encode("refund policy international")
doc_vector = bi_encoder.encode("Refund requests must be submitted within 30 days...")
# Similarity is based on compressed representations
similarity = cosine_similarity(query_vector, doc_vector) # 0.76Why This Fails
- Information loss: Compressing a document into 768 dimensions loses nuance
- No interaction: Query and document are encoded independently
- Approximate matching: Similar vectors ≠ relevant documents
⚠️ The Core Issue: Bi-encoders must compress all possible meanings of a document into a single vector. This makes them fast but imprecise for nuanced queries.
The Solution: Two-Stage Retrieval
The solution is simple: use a fast model for initial retrieval, then a precise model for reranking.
# Stage 1: Fast retrieval (bi-encoder)
results = vector_store.search(query, k=20) # Get top 20
# Stage 2: Precise reranking (cross-encoder)
reranked = cross_encoder.rerank(query, results, top_k=5) # Keep top 5
Bi-Encoder: Fast but Approximate
How It Works
Bi-encoders encode query and document separately:
# Bi-encoder architecture
class BiEncoder:
def __init__(self):
self.encoder = BertModel.from_pretrained("bert-base")
def encode_query(self, query):
return self.encoder(query).pooler_output # [768]
def encode_document(self, doc):
return self.encoder(doc).pooler_output # [768]
def similarity(self, query, doc):
q_vec = self.encode_query(query)
d_vec = self.encode_document(doc)
return cosine_similarity(q_vec, d_vec)Advantages
- Fast: Documents can be pre-encoded and indexed
- Scalable: Search is O(1) with approximate nearest neighbors
- Efficient: Single forward pass per document
Disadvantages
- Information loss: Compression loses nuance
- No interaction: Query and document never "see" each other
- Approximate: Similar vectors may not be relevant
Cross-Encoder: Slow but Precise
How It Works
Cross-encoders process query and document together with full attention:
# Cross-encoder architecture
class CrossEncoder:
def __init__(self):
self.encoder = BertModel.from_pretrained("cross-encoder/ms-marco")
def score(self, query, document):
# Joint input with special tokens
input_text = f"[CLS] {query} [SEP] {document} [SEP]"
# Full attention between query and document
outputs = self.encoder(input_text)
# Relevance score
relevance = self.classifier(outputs.pooler_output)
return relevance.item() # 0.97Why It's More Accurate
- Full attention: Query and document tokens attend to each other
- No compression: Works with raw token representations
- Joint encoding: Captures fine-grained relevance
💡 Key Insight: Cross-encoders can distinguish between "refund policy" (relevant) and "general returns" (less relevant) because they see the full interaction between query and document tokens.
Bi-Encoder vs Cross-Encoder
| Aspect | Bi-Encoder | Cross-Encoder |
|---|---|---|
| Architecture | Separate encoders | Joint encoder |
| Speed | Fast (pre-computed) | Slow (per-pair) |
| Accuracy | Good (approximate) | Excellent (precise) |
| Use Case | Initial retrieval | Reranking |
| Pre-computation | Yes (embed once) | No (compute per query) |
| Typical K | 100-1000 | 10-50 |
| Latency | 1-10ms | 50-200ms per pair |
Conceptual Python Example
Here's how to implement reranking in a RAG pipeline:
from sentence_transformers import CrossEncoder
import chromadb
# Initialize components
cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
chroma_client = chromadb.Client()
collection = chroma_client.create_collection("documents")
def rag_with_reranking(query: str, k_initial: int = 20, k_final: int = 5):
"""
Two-stage retrieval: fast retrieval + precise reranking
"""
# Stage 1: Fast retrieval with bi-encoder (vector search)
results = collection.query(
query_texts=[query],
n_results=k_initial
)
documents = results['documents'][0]
metadatas = results['metadatas'][0]
# Stage 2: Precise reranking with cross-encoder
# Create (query, document) pairs
pairs = [(query, doc) for doc in documents]
# Score each pair
scores = cross_encoder.predict(pairs)
# Sort by relevance score
scored_results = list(zip(documents, metadatas, scores))
scored_results.sort(key=lambda x: x[2], reverse=True)
# Return top-k
reranked = scored_results[:k_final]
return reranked
# Example usage
query = "What is the refund policy for international orders?"
results = rag_with_reranking(query)
for doc, metadata, score in results:
print(f"Score: {score:.3f} | Source: {metadata['source']}")
print(f"Content: {doc[:100]}...")
print()Expected Output
Score: 0.973 | Source: refund_policy.pdf
Content: Refund requests must be submitted within 30 days of purchase...
Score: 0.842 | Source: return_policy.pdf
Content: Our general return policy allows returns within...
Score: 0.718 | Source: international_shipping.pdf
Content: International shipping rates vary by destination...Popular Reranking Models
| Model | Type | Speed | Quality | Best For |
|---|---|---|---|---|
| cross-encoder/ms-marco-MiniLM | Local | Fast | Good | Prototyping |
| BGE-reranker-v2-m3 | Local | Medium | Excellent | Production |
| Cohere Rerank | API | Fast | Excellent | Managed solution |
| Jina Reranker | API/Local | Fast | Excellent | Multilingual |
| FlashRank | Local | Very fast | Good | Low latency |
Performance Impact
Reranking consistently improves retrieval quality:
| Metric | Without Reranking | With Reranking | Improvement |
|---|---|---|---|
| Precision@5 | 0.65 | 0.85 | +30% |
| Recall@10 | 0.78 | 0.92 | +18% |
| MRR (Mean Reciprocal Rank) | 0.72 | 0.91 | +26% |
| NDCG@5 | 0.68 | 0.88 | +29% |
💡 Research Findings: Studies show cross-encoder reranking improves RAG accuracy by 20-40% on average, with some datasets showing improvements up to 60%.
When to Add Reranking
Add Reranking When:
- Initial retrieval returns irrelevant results
- Precision is more important than speed
- You have a small-to-medium candidate set (10-50 docs)
- Query-document relevance is nuanced
- You're building a production system
Skip Reranking When:
- Speed is critical (real-time applications)
- Document collection is very small (< 100 docs)
- Queries are simple keyword matches
- Latency budget is very tight (< 50ms total)
Decision Framework
if candidate_count > 20:
# Use reranking
results = retrieve(query, k=20)
results = rerank(query, results, k=5)
elif precision_critical:
# Use reranking even for small sets
results = retrieve(query, k=10)
results = rerank(query, results, k=5)
else:
# Skip reranking for speed
results = retrieve(query, k=5)Production Reranking Architecture
class ProductionRAG:
def __init__(self):
self.vector_store = ChromaDB()
self.cross_encoder = CrossEncoder("BGE-reranker-v2-m3")
self.llm = Ollama(model="llama3.1:8b")
def query(self, question: str) -> dict:
# Stage 1: Fast retrieval
candidates = self.vector_store.search(question, k=20)
# Stage 2: Reranking
reranked = self.rerank(question, candidates, top_k=5)
# Stage 3: Context construction
context = self.build_context(reranked)
# Stage 4: LLM generation
answer = self.llm.generate(
f"Context: {context}\n\nQuestion: {question}\n\nAnswer:"
)
# Stage 5: Citations
sources = [r['metadata'] for r in reranked]
return {"answer": answer, "sources": sources}
def rerank(self, query, candidates, top_k=5):
pairs = [(query, c['text']) for c in candidates]
scores = self.cross_encoder.predict(pairs)
for i, score in enumerate(scores):
candidates[i]['rerank_score'] = float(score)
candidates.sort(key=lambda x: x['rerank_score'], reverse=True)
return candidates[:top_k]Try It Yourself
Add reranking to your RAG system with these BestWordz resources:
Conclusion
Vector search alone is not enough for production RAG. Here's why:
- Bi-encoders are fast but lose information through compression
- Cross-encoders are slow but capture fine-grained relevance
- Two-stage retrieval gives you the best of both worlds
The pattern is simple:
Stage 1: Retrieve top 20 with bi-encoder (fast)
Stage 2: Rerank to top 5 with cross-encoder (precise)
Stage 3: Generate answer with top 5 (grounded)Reranking typically improves precision by 20-40%. For any production RAG system, it's the highest-ROI improvement you can make.