🎯 Key Takeaway: RAG (Retrieval-Augmented Generation) grounds LLM responses in your actual documents. Every component — from ingestion to citations — matters for production quality. There are 9 critical components, and weakness in any one degrades the entire system.
You ask an AI assistant: "What is our company's refund policy?"
Without RAG, the LLM might:
- Make up a plausible but incorrect policy
- Use training data from other companies
- Give a generic answer that doesn't match your actual policy
With RAG, the system:
- Searches your actual policy documents
- Retrieves the relevant sections
- Generates an answer grounded in those documents
- Cites the specific source
This article explains every component of a RAG architecture — how they work, why they matter, and what can go wrong.
The 9 Components of RAG
| # | Component | Input | Output |
|---|---|---|---|
| 1 | Ingestion | Raw documents | Parsed text |
| 2 | Chunking | Parsed text | Text chunks |
| 3 | Embedding | Text chunks | Vector embeddings |
| 4 | Vector Store | Embeddings + metadata | Indexed database |
| 5 | Retrieval | Query embedding | Top-K chunks |
| 6 | Reranking | Retrieved chunks | Reranked chunks |
| 7 | Context Construction | Reranked chunks | Prompt with context |
| 8 | LLM Generation | Prompt + context | Generated answer |
| 9 | Citations | Answer + sources | Answer with citations |
Component Details
1. Ingestion: Getting Documents into the System
Ingestion converts raw documents into processable text. This sounds simple, but document formats are complex:
| Format | Challenge | Tool |
|---|---|---|
| Layout, tables, images | PyPDF, Unstructured | |
| DOCX | Formatting, tables | python-docx |
| HTML | Navigation, ads, scripts | BeautifulSoup, Trafilatura |
| Markdown | Headings, code blocks | Direct parsing |
| CSV/Excel | Structured data | pandas |
Best Practices
- Extract text, not just raw bytes
- Preserve document structure (headings, paragraphs)
- Handle metadata (title, author, date)
- Clean up artifacts (headers, footers, page numbers)
2. Chunking: Splitting Documents into Pieces
Documents are too large to embed whole. Chunking splits them into manageable pieces while preserving context.
Chunking Strategies
| Strategy | How It Works | Best For |
|---|---|---|
| Fixed-size | Split every N characters | Simple documents |
| Sentence-based | Split at sentence boundaries | Narrative text |
| Recursive | Split at headers, then paragraphs | Structured documents |
| Semantic | Split when meaning changes | Complex topics |
| Structure-aware | Respect document structure | Technical docs |
Key Parameters
# Chunk size: 1000 characters (typical)
# Overlap: 200 characters (preserves context)
# Separators: ["\n\n", "\n", ". ", " ", ""]
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
separators=["\n\n", "\n", ". ", " ", ""]
)
⚠️ Chunk Size Matters:
• Too small (100 chars) → Loses context, retrieval is imprecise
• Too large (5000+ chars) → Retrieves irrelevant information
• Sweet spot: 500-1500 chars with 10-20% overlap
• Too small (100 chars) → Loses context, retrieval is imprecise
• Too large (5000+ chars) → Retrieves irrelevant information
• Sweet spot: 500-1500 chars with 10-20% overlap
4. Vector Store: Indexing and Searching
The vector store indexes embeddings for fast similarity search. When a query comes in, it finds the most relevant chunks.
Vector Store Options
| Store | Type | Best For |
|---|---|---|
| ChromaDB | Local/Embedded | Prototyping, small-medium datasets |
| FAISS | Library | High-performance local search |
| Pinecone | Cloud SaaS | Production, managed infrastructure |
| Weaviate | Self-hosted/Cloud | Hybrid search, complex queries |
| Qdrant | Self-hosted/Cloud | Performance, filtering |
Similarity Metrics
# Cosine similarity (most common)
similarity = dot(a, b) / (norm(a) * norm(b))
# Range: -1 to 1 (1 = identical)
# Euclidean distance
distance = sqrt(sum((a - b)²))
# Range: 0 to ∞ (0 = identical)
# Dot product
similarity = dot(a, b)
# Range: -∞ to ∞5. Retrieval: Finding Relevant Chunks
Retrieval searches the vector store for chunks relevant to the user's query.
Retrieval Strategies
| Strategy | How It Works | Pros/Cons |
|---|---|---|
| Semantic Search | Vector similarity only | Good for meaning, misses keywords |
| Keyword Search (BM25) | Term frequency matching | Exact matches, no semantics |
| Hybrid Search | Combine semantic + keyword | Best of both worlds |
Top-K Selection
# Retrieve top 4 most relevant chunks
results = vector_store.similarity_search(
query,
k=4 # Number of chunks to retrieve
)
# Too few (k=1): May miss important context
# Too many (k=20): Retrieves irrelevant information
# Sweet spot: k=4 to k=106. Reranking: Improving Precision
Initial retrieval is fast but imprecise. Reranking uses a more powerful model to reorder results by relevance.
Why Rerank?
# Initial retrieval (fast, approximate)
Top 10 chunks by cosine similarity
# Reranking (slower, precise)
Cross-encoder scores each (query, chunk) pair
Reorders by actual relevance
# Result: Top 4 most relevant chunksReranking Models
| Model | Type | Quality |
|---|---|---|
| Cohere Rerank | API | Excellent |
| cross-encoder/ms-marco | Local | Good |
| BGE Reranker | Local | Excellent |
💡 Impact: Reranking typically improves retrieval precision by 20-40%. It's optional but highly recommended for production systems.
7. Context Construction: Building the Prompt
Context construction assembles retrieved chunks into a prompt the LLM can use.
Prompt Template
template = """Use the following context to answer the question.
If you cannot find the answer, say "I don't have enough information."
Context:
{context}
Question: {question}
Answer:"""
# Context is constructed from retrieved chunks
context = "\n\n".join([chunk.text for chunk in reranked_chunks])Context Window Management
| LLM | Context Window | Approx. Chunks |
|---|---|---|
| Llama 3.1 8B | 8K tokens | ~6 chunks |
| GPT-4o | 128K tokens | ~100 chunks |
| Claude 3.5 | 200K tokens | ~150 chunks |
⚠️ Context Overflow: If context exceeds the LLM's window, you'll get errors or truncated responses. Always check token counts.
8. LLM Generation: Creating the Answer
The LLM generates an answer using the retrieved context as grounding.
Generation Parameters
response = llm.generate(
prompt,
temperature=0.3, # Low = more grounded
max_tokens=1024,
top_p=0.9
)
# Temperature effects:
# 0.0 = Deterministic, factual
# 0.3 = Slightly creative, still grounded
# 0.7 = More creative, risk of hallucination
# 1.0 = Very creative, unreliable for RAGHallucination Mitigation
- Use low temperature (0.0-0.3)
- Explicitly instruct: "Use only the provided context"
- Include citations in the prompt
- Validate outputs against sources
9. Citations: Source Attribution
Citations link answers back to source documents, enabling verification.
Citation Approaches
| Approach | Example | Best For |
|---|---|---|
| Inline | "Policy requires 30-day notice [1]" | Simple answers |
| Footnotes | "Policy requires 30-day notice. Sources: [1] Policy.pdf, p.3" | Detailed answers |
| Confidence | "Based on 3 sources (high confidence)" | Trust scoring |
Metadata for Citations
# Store metadata with each chunk
metadata = {
"source": "policy.pdf",
"page": 3,
"section": "Refund Policy",
"last_updated": "2026-01-15"
}
# Include in citations
"According to the Refund Policy (policy.pdf, p.3)..."End-to-End Example
Here's how all components work together:
# 1. Ingestion
documents = load_documents("./docs/")
# 2. Chunking
chunks = text_splitter.split_documents(documents)
# 3. Embedding
embeddings = embedding_model.embed(chunks)
# 4. Vector Store
vector_store.add(chunks, embeddings)
# 5. Retrieval
query_embedding = embedding_model.embed("What is the refund policy?")
results = vector_store.search(query_embedding, k=10)
# 6. Reranking
reranked = reranker.rerank(query, results, top_k=4)
# 7. Context Construction
context = "\n\n".join([r.text for r in reranked])
prompt = f"Context: {context}\n\nQuestion: {query}\n\nAnswer:"
# 8. LLM Generation
answer = llm.generate(prompt, temperature=0.3)
# 9. Citations
sources = [r.metadata for r in reranked]
final_answer = format_with_citations(answer, sources)Try It Yourself
Build your own RAG system with these BestWordz resources:
Common RAG Mistakes
| Mistake | Impact | Solution |
|---|---|---|
| Chunks too large | Retrieves irrelevant info | Use 500-1500 char chunks |
| No overlap | Loses context at boundaries | Add 10-20% overlap |
| Wrong embedding model | Poor semantic matching | Test multiple models |
| No reranking | Lower precision | Add reranker for production |
| High temperature | Hallucination | Use 0.0-0.3 for RAG |
| No citations | Unverifiable answers | Always include sources |
RAG Production Checklist
| Component | Check | Status |
|---|---|---|
| Ingestion | Handles all document formats? | ☑️ |
| Chunking | Appropriate chunk size and overlap? | ☑️ |
| Embedding | Tested multiple models? | ☑️ |
| Vector Store | Scalable for your data size? | ☑️ |
| Retrieval | Hybrid search enabled? | ☑️ |
| Reranking | Added for production? | ☑️ |
| Context | Within LLM token limit? | ☑️ |
| LLM | Low temperature (0.0-0.3)? | ☑️ |
| Citations | Source attribution included? | ☑️ |
Conclusion
RAG is not just "add a vector store." It's a complete pipeline with 9 interconnected components:
- Ingestion — Get documents into the system
- Chunking — Split into processable pieces
- Embedding — Convert to vectors
- Vector Store — Index for fast search
- Retrieval — Find relevant chunks
- Reranking — Improve precision
- Context Construction — Build the prompt
- LLM Generation — Create the answer
- Citations — Link to sources
Each component matters. Weakness in any one degrades the entire system. For production RAG:
- Start with proven chunking strategies (recursive, 1000 chars)
- Test multiple embedding models
- Add reranking for precision
- Use low temperature for grounded answers
- Always include citations
- Evaluate end-to-end, not just retrieval