RAG architecture patterns that survive production

The short version

Retrieval-augmented generation is the unglamorous backbone of most production LLM apps: instead of hoping the model memorized your data, you fetch the relevant documents at query time and stuff them into the prompt. The model stays general; your data stays fresh. Almost every "AI over our docs" product you've used is RAG wearing a trench coat.

The pattern is simple to describe and easy to get subtly wrong. The difference between a RAG system that feels magical and one that hallucinates with citations is almost entirely in the retrieval half.

The pipeline, piece by piece

A production RAG system has five stages, and each one is a place to win or lose:

1. Ingestion. Documents get chunked, cleaned, and embedded. Chunking strategy matters more than embedding model choice: 400–800 token chunks with ~10% overlap is the boring default that works. Respect document structure — never split a table across chunks if you can avoid it.
2. Indexing. Embeddings land in a vector index alongside metadata filters (date, source, access level). Metadata is how you answer "only search docs from after the migration" without retraining anything.
3. Retrieval. The query is embedded, the top-k chunks come back. Hybrid search — dense vectors plus BM25 keyword matching, fused with reciprocal rank — beats pure vector search on almost every real workload, especially ones with product names, error codes, and version numbers.
4. Reranking. A cross-encoder scores the top 30–50 candidates and keeps the best 5–8. This is the highest-ROI step most teams skip. It costs one extra model call and routinely fixes "the right doc was #14" problems.
5. Generation. The surviving chunks go into the prompt with explicit instructions to cite and to say "I don't know" when the context doesn't cover the question.

The failure modes you'll actually hit

• Chunking destroyed the answer. The key sentence sits at a chunk boundary and lost its context. Fix with overlap and structure-aware splitting.
• Stale index. Docs updated, embeddings didn't. Version your index and rebuild on a schedule, or better, on document change events.
• The model ignores the context. Usually a prompting problem: put the retrieved text first, the question last, and add "answer only from the context below."
• Evaluation theater. Teams eval the generator and ignore the retriever. Measure retrieval recall@k on real questions first — if the right chunk isn't retrieved, no prompt engineering will save you.

Rule of thumb: spend 70% of your RAG effort on retrieval quality (chunking, hybrid search, reranking) and 30% on the generation prompt. Most teams do the opposite.