RAG Architecture: Industry-Standard Design

Overview

RAG (Retrieval-Augmented Generation) combines retrieval of relevant documents with language model generation. This document covers the complete architecture used in production systems.

Core Components

1. Document Ingestion Pipeline

Purpose: Process and store documents for retrieval

Steps:

a) Document Loading:

PDF, DOCX, HTML, Markdown, etc.
Web scraping
Database queries
API integrations

b) Text Extraction:

OCR for scanned documents
Table extraction
Code extraction
Metadata extraction

c) Preprocessing:

Clean text (remove headers, footers)
Normalize whitespace
Handle special characters
Language detection

d) Chunking:

Split documents into smaller pieces
Multiple strategies (see chunking strategies)

e) Metadata Extraction:

Document source
Creation date
Author
Section titles
Page numbers

2. Embedding Generation

Purpose: Convert text chunks into vector representations

Components:

a) Embedding Model:

Text embeddings: sentence-transformers, OpenAI embeddings
Multilingual: mBERT, multilingual models
Domain-specific: Fine-tuned on domain data

b) Embedding Strategy:

Single embedding: One vector per chunk
Multi-vector: Multiple embeddings per chunk (different granularities)
Hybrid: Dense + sparse (BM25) embeddings

c) Embedding Storage:

Vector database (Pinecone, Weaviate, Qdrant, Chroma)
Metadata filtering
Index optimization

3. Query Processing

Purpose: Process user queries for retrieval

Steps:

a) Query Understanding:

Intent detection
Entity extraction
Query expansion
Query rewriting

b) Embedding Generation:

Same embedding model as documents
Query-specific preprocessing

c) Hybrid Search:

Dense retrieval (vector similarity)
Sparse retrieval (keyword matching, BM25)
Combine both

4. Retrieval System

Purpose: Find relevant documents for query

Components:

a) Sparse Retrieval (BM25, TF-IDF):

BM25: Industry standard for keyword-based retrieval
- Term frequency saturation
- Document length normalization
- Better than TF-IDF
TF-IDF: Simple, interpretable
Use: Exact term matching, keywords

b) Dense Retrieval (Embeddings):

Vector similarity (cosine, dot product)
Semantic understanding
Use: Related concepts, synonyms

c) Hybrid Search:

Combine sparse + dense
Weighted combination: α × BM25 + (1-α) × Dense
Use: Production systems (best of both)

d) Vector Search:

Approximate nearest neighbor (ANN) search
FAISS, HNSW, IVF
Fast retrieval for large datasets

e) Filtering:

Metadata filters (date, source, type)
Access control
Relevance thresholds

f) Re-ranking:

Cross-encoder for better accuracy
Learning-to-rank models
Multi-stage retrieval

See retrieval_methods.md and retrieval_implementations.py for detailed implementations!

5. Context Assembly

Purpose: Prepare context for generation

Steps:

a) Chunk Selection:

Top-K retrieval results
Diversity selection
Deduplication

b) Context Ordering:

Relevance-based ordering
Chronological ordering
Hierarchical ordering

c) Context Truncation:

Fit within model context window
Priority-based truncation
Summary for overflow

6. Generation

Purpose: Generate answer using retrieved context

Components:

a) Prompt Engineering:

System prompts
Context formatting
Few-shot examples
Instructions

b) Generation Parameters:

Temperature
Top-p sampling
Max tokens
Stop sequences

c) Post-processing:

Answer extraction
Citation generation
Confidence scoring
Hallucination detection

Complete RAG Pipeline

User Query
    ↓
Query Processing (embedding, expansion)
    ↓
Retrieval (vector search + filtering)
    ↓
Re-ranking (optional, for accuracy)
    ↓
Context Assembly (top-K chunks, ordering)
    ↓
Generation (LLM with context)
    ↓
Post-processing (extraction, citations)
    ↓
Final Answer