RAG Architecture Explained

What RAG is

Retrieval-augmented generation pairs a language model with an external knowledge source. The model retrieves relevant documents at query time and grounds its answer in them, instead of relying only on the weights learned during training. The architecture and the name come from Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", published at NeurIPS 2020 (arXiv:2005.11401). The original system combined a BART sequence-to-sequence generator with a dense passage retrieval index over Wikipedia, and it defined two formulations: RAG-Sequence, which conditions the whole generated answer on one set of retrieved documents, and RAG-Token, which can draw on different documents for different tokens.

The practical point is simple. Training data goes stale, and fine-tuning is expensive for every knowledge update. RAG makes knowledge updates a data operation: add or change documents in the index, and the answers change without retraining the model.

The indexing stage

Indexing happens offline and only once per corpus change. Documents are loaded, split into chunks, converted to vectors by an embedding model, and stored in a vector index alongside the raw text and metadata.

Chunking decides the granularity of everything downstream. A chunk that is too large dilutes the embedding with mixed topics, and retrieval returns noise. A chunk that is too small loses the context needed to answer. Most production pipelines use chunks of a few hundred tokens with a small overlap between neighbors, tuned per corpus.

Embedding turns text into a fixed-length numeric vector, usually 384 to 3072 dimensions depending on the model. Common choices include the BGE family, E5, and OpenAI's text-embedding models. The query will be embedded with the same model, so the index is locked to one embedding model at a time. Changing models means re-embedding the whole corpus.

Metadata stored next to each vector is what makes retrieval precise later: source file, page number, section heading, date, author. Filtering on metadata before or during vector search removes most retrieval junk before it reaches the model.

The retrieval stage

A user query arrives and is embedded with the same model used at indexing. The vector database runs an approximate nearest neighbor search, usually over an HNSW or IVF index, and returns the top k chunks ranked by a distance metric. Cosine similarity is the default for text embeddings:

```python

import numpy as np

def cosine_similarity(a, b):

a, b = np.asarray(a), np.asarray(b)

return float(a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))

scores = [cosine_similarity(query_vec, chunk_vec) for chunk_vec in index]

top_k = sorted(zip(chunk_ids, scores), key=lambda x: x[1], reverse=True)[:5]

```

Pure dense retrieval misses exact terms, so most systems add lexical search with BM25 alongside the dense search and merge the two rankings with reciprocal rank fusion. A cross-encoder reranker then scores the merged candidates directly against the query and keeps the strongest few. The retriever-plus-reranker pair is the single biggest lever on RAG answer quality.

Retrieval is parameterized by k, the number of chunks returned. Larger k raises the chance the answer is present but adds noise and cost, and long contexts degrade model accuracy on middle positions. Typical starting values are 20 to 50 candidates from the first pass, reranked down to 5 to 10 for the prompt.

The augmentation and generation stages

The selected chunks are formatted into the prompt with citations or source labels, the user question is appended, and the model generates the answer. A working template is:

```text

Answer the question using only the context below.

Cite the source of each factual claim as [source: <id>].

Context:

[1] <chunk text> (source: handbook.pdf, p. 13)

[2] <chunk text> (source: wiki.md)

Question: <user question>

Answer:

```

Generation temperature stays low, usually 0 to 0.3, because retrieval has already supplied the facts and the model's job is synthesis, not invention. Post-processing checks that citations resolve to real chunk ids and that the answer does not introduce claims absent from the context.

Advanced patterns

Query rewriting uses a small model call to expand or clarify the query before retrieval, which helps with vague or multi-hop questions. HyDE generates a hypothetical answer from the query, embeds that answer, and retrieves with it, which works because the answer-like text is closer in vector space to the target chunks than the question is.

Corrective and agentic patterns add a loop: the system grades retrieved chunks for relevance, rewrites the query and retries when they are poor, or calls retrieval repeatedly during generation when the model emits a low-confidence token. These trade latency for accuracy on hard questions.

Where RAG fails first

Most failures trace to retrieval, not generation. The index contains the answer but the query embedding does not land near it, chunk boundaries split the relevant fact across two chunks, or the corpus has contradictory documents and the model cannot arbitrate between them. The fix order that works in practice: inspect the retrieved chunks for the failing queries first, then adjust chunking and metadata, then the embedding model, then reranking, and only then the prompt. Measuring retrieval with recall at k on a labeled set turns this from guesswork into a number that moves.