RAG Architecture

What it is

Retrieval-Augmented Generation (RAG) is an architecture that enhances large language model responses by first retrieving relevant documents or passages from an external knowledge base and injecting them into the model's context window before generating a response. It combines the broad language understanding and generation capabilities of LLMs with the ability to ground responses in specific, up-to-date, private, or domain-specific information that was not part of the model's training data.

The standard RAG pipeline operates in two phases. The indexing phase (offline) ingests documents, splits them into chunks of appropriate size, computes dense vector embeddings for each chunk using an embedding model, and stores those vectors in a vector database alongside the original chunk text. The querying phase (online) receives a user question, embeds the question using the same embedding model, performs an approximate nearest neighbor (ANN) search in the vector database to retrieve the most semantically similar chunks, constructs a prompt that includes both the retrieved context and the original question, and passes this augmented prompt to the LLM for generation.

Advanced RAG architectures extend this basic pattern in several ways: hybrid search combines dense vector search with traditional BM25 keyword search for better recall; re-ranking applies a cross-encoder model to the top-K retrieved chunks to more accurately score their relevance before passing them to the LLM; query decomposition breaks complex questions into sub-questions retrieved independently; and iterative retrieval (or agentic RAG) performs multiple retrieval steps interleaved with generation when a single retrieval round is insufficient.

Why it exists

LLMs have a fixed knowledge cutoff date — they don't know about events that occurred after their training data was collected. They also don't have access to private enterprise data (internal documents, proprietary databases, customer records). Extending an LLM's knowledge by retraining or fine-tuning on new data is prohibitively expensive and time-consuming for most organizations. RAG provides a cost-effective alternative: the model's parametric knowledge stays fixed, but it is augmented at inference time with retrieved information that is always current.

LLMs also hallucinate — they generate plausible-sounding but factually incorrect statements when operating purely from their parametric memory, especially for specific facts, figures, and citations. RAG grounds generation in retrieved source documents, dramatically reducing hallucination on verifiable factual questions and providing the ability to cite sources. When the retrieval step returns relevant, accurate documents, the LLM acts more as a synthesizer and formatter of that information than as a memory recall mechanism.

Fine-tuning an LLM on domain-specific data changes the model's behavior broadly but doesn't give it the ability to provide specific, current answers with citations. Fine-tuning is most effective for adapting the model's style, format, and domain vocabulary, not for injecting large amounts of factual knowledge. RAG and fine-tuning are complementary: fine-tune for style and domain adaptation, use RAG for grounded knowledge retrieval.

When to use

• Question-answering over a large private document corpus (internal knowledge bases, documentation, legal documents, support tickets).
• Applications that require up-to-date information beyond the LLM's training cutoff (news, recent research papers, product catalogs).
• Use cases where citations and source attribution are required — RAG can return the exact passages that informed a response.
• Reducing hallucination in factual question-answering by grounding the model's responses in retrieved evidence.
• Cost-effective knowledge expansion for domain-specific applications without the expense of continuous full retraining.

When NOT to use

• Tasks that are purely generative and don't require factual grounding — creative writing, summarization of provided text, code generation — where retrieval adds latency without benefit.
• When the knowledge base is small enough (<100 documents) to fit entirely in a long-context LLM's context window — full context may outperform retrieval.
• When the query is highly ambiguous or requires multi-hop reasoning across many documents — retrieval quality degrades quickly for complex queries without sophisticated re-ranking and query decomposition.

Typical architecture

 INDEXING PIPELINE (offline / batch)
 ┌──────────────────────────────────────────────────────────┐
 │  Document Sources                                         │
 │  [PDFs] [Confluence] [SharePoint] [Databases] [Web]      │
 └──────────────────────┬───────────────────────────────────┘
                        │
                        ▼
 ┌──────────────────────────────────────────────────────────┐
 │  DOCUMENT PROCESSOR                                       │
 │  Extract text · Clean / normalize · Metadata tagging      │
 └──────────────────────┬───────────────────────────────────┘
                        │
                        ▼
 ┌──────────────────────────────────────────────────────────┐
 │  CHUNKING STRATEGY                                        │
 │  Fixed-size · Sentence boundary · Semantic · Hierarchical│
 │  Chunk: 256–512 tokens + overlap (50 tokens)             │
 └──────────────────────┬───────────────────────────────────┘
                        │ chunks
                        ▼
 ┌──────────────────────────────────────────────────────────┐
 │  EMBEDDING MODEL                                          │
 │  text-embedding-ada-002 · BGE · E5 · Cohere Embed         │
 │  Chunk text → 768/1536-dim dense vector                   │
 └──────────────────────┬───────────────────────────────────┘
                        │ (chunk_text, embedding, metadata)
                        ▼
 ┌──────────────────────────────────────────────────────────┐
 │  VECTOR DATABASE                                          │
 │  Pinecone · Weaviate · Qdrant · pgvector · Chroma        │
 │  ANN index (HNSW) + metadata filters                     │
 └──────────────────────────────────────────────────────────┘

 QUERY PIPELINE (online / per-request)
 ┌──────────┐    ┌─────────────┐    ┌────────────────────┐
 │  User    │───▶│  Query      │───▶│  Vector DB         │
 │  Query   │    │  Embedding  │    │  ANN Search top-K  │
 └──────────┘    └─────────────┘    └────────┬───────────┘
                                             │  top-K chunks
                                             ▼
                                   ┌────────────────────┐
                                   │   RE-RANKER         │
                                   │  (optional)         │
                                   │  Cross-encoder      │
                                   │  Reorder top-K      │
                                   └────────┬───────────┘
                                            │ top-N chunks
                                            ▼
 ┌──────────────────────────────────────────────────────┐
 │  PROMPT ASSEMBLY                                      │
 │  System: "Answer using context below..."             │
 │  Context: [chunk1] [chunk2] [chunk3]                 │
 │  Question: [user query]                              │
 └──────────────────────────────┬───────────────────────┘
                                │
                                ▼
 ┌──────────────────────────────────────────────────────┐
 │  LLM GENERATION                                       │
 │  Response with citations                             │
 └──────────────────────────────────────────────────────┘

Pros and cons

Implementation notes

Chunking strategy has an outsized impact on retrieval quality and is often underestimated. Fixed-size chunking (e.g., 512 tokens with 50-token overlap) is simple but can split sentences and break semantic coherence. Sentence-boundary or paragraph-boundary chunking preserves semantic units but produces variable-length chunks. Hierarchical chunking (storing both fine-grained and coarse-grained representations) supports a "small-to-big" retrieval strategy where small chunks are retrieved for precision but larger surrounding context is passed to the LLM. The optimal chunk size is use-case dependent — experiment empirically using retrieval evaluation metrics (MRR, NDCG, recall@K) on a representative question set.

Embed both the query and the chunks using the same embedding model, and be aware that embedding models have their own context length limits (typically 512–8192 tokens). If chunks exceed the embedding model's context window, they will be silently truncated, degrading retrieval quality for long passages. Evaluate embedding models on your specific domain and language — general-purpose models may underperform on highly technical domains. Domain-specific fine-tuned embedding models (available for legal, medical, and code domains) often significantly outperform general-purpose embeddings.

Implement a retrieval evaluation framework before deploying to production. Create a "golden dataset" of 100-200 representative questions paired with the expected source documents. Measure recall@K (what fraction of expected source documents appear in the top-K retrieved chunks) and MRR (mean reciprocal rank) against this dataset for each configuration change. Without quantitative retrieval evaluation, teams end up guessing at chunk sizes, overlap, and embedding models rather than making evidence-based choices. LlamaIndex and LangChain both include evaluation utilities for this purpose.

Common failure modes

• Lost in the middle: LLMs are known to underweight information in the middle of a long context, attending more to the beginning and end. If the most relevant chunk is ranked fifth of ten and placed in the middle of the prompt, the model may ignore it. Reranking to surface the most relevant chunks first and limiting context to the top 3-5 chunks mitigates this.
• Semantic mismatch between query and indexed content: User questions use different vocabulary than the indexed documents (e.g., users ask about "heart attack" but documents use "myocardial infarction"). Hybrid search (combining dense retrieval with BM25 keyword matching) and query expansion techniques address vocabulary mismatch.
• Stale index: Documents are updated or deleted in the source system but the vector index is not updated accordingly, causing the LLM to cite outdated or retracted information. Implement incremental indexing with source change detection and TTL-based re-indexing for mutable document sources.
• Context window overflow: Retrieving too many chunks or overly large chunks fills the LLM's context window, truncating either the retrieved context or the original question. Always calculate the token count of the assembled prompt before sending to the LLM and implement a dynamic chunk count limit based on available context tokens.

Decision checklist

• What chunking strategy and chunk size will you use, and have you evaluated retrieval quality metrics against a golden dataset?
• Which embedding model is appropriate for your domain — general-purpose or domain-fine-tuned?
• Is hybrid search (dense + BM25) necessary to handle vocabulary mismatch in your corpus?
• Do you need a re-ranking step, and have you measured its latency impact against the improvement in answer quality?
• How will you handle document updates and deletions in the source — is incremental indexing implemented?
• Is there a retrieval evaluation framework to objectively measure and improve retrieval quality over time?

Example use cases

• Internal knowledge base assistant: Indexes Confluence pages, engineering runbooks, and Slack archives; answers employee questions about internal processes with citations to the exact pages, updated whenever documents are edited.
• Legal contract analysis: Ingests thousands of contracts; lawyers ask questions like "which contracts contain automatic renewal clauses with less than 30 days notice?"; retrieval surfaces the relevant contract sections; the LLM extracts and summarizes the relevant clause text.
• Technical support chatbot: Indexes product documentation, known issue articles, and support ticket histories; hybrid retrieval handles both precise keyword queries and semantically fuzzy user descriptions of problems; re-ranker ensures the most relevant articles are surfaced first.

Related patterns

• Vector Databases
• LLM Architecture
• Search & Retrieval
• Feature Stores

Further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks – Lewis et al. (original RAG paper)
• LlamaIndex Documentation – Building RAG Applications
• LangChain – Question Answering over Documents
• Retrieval Augmented Generation – Pinecone Learning Center