RAG stands for Retrieval Augmented Generation. The name is technical. The idea is simple: before the model answers, go find the relevant information and put it in the context window. Let the model read it. Then let the model answer.
Imagine you have a brilliant analyst on your team. They're great at synthesizing information, writing clearly, drawing conclusions, and explaining complex things. But they don't have deep knowledge of your specific company, your customers, or your internal systems.
You could spend months training them on everything your organization knows. Or you could do something simpler: when you need them to answer a specific question, have someone pull the relevant files first, hand those files to the analyst, and let the analyst work from those.
That's RAG. The "retrieval" step pulls the relevant documents. The "augmented" part means those documents get added to the context. The "generation" is the model doing what it does — reading the context and producing a response. The model's general intelligence does the synthesis. The retrieved documents provide the specific, current, private knowledge.
The model itself is completely unchanged. Same weights, same architecture, same frozen knowledge. What changes is what arrives in the inbox before the model reads it. From the model's perspective, it simply received a longer, richer context. It has no awareness that the extra content was retrieved automatically. It just reads everything it's given and responds.
The retrieval step needs to find the right documents from potentially thousands or millions of possibilities — fast enough to feel instantaneous. It does this not by searching for exact keyword matches (though that helps), but by searching for meaning similarity.
Remember vectors from the Overview track — the lists of numbers that represent where a token sits in meaning-space? The same idea applies to whole chunks of text. Every document in your knowledge base gets converted into a vector that represents its meaning. Your question gets converted into a vector too. The retrieval step finds the documents whose vectors are closest to your question's vector — the documents that are most about what you're asking, even if they don't use the exact same words.
This is called semantic search — searching by meaning rather than by keyword. It's why RAG can surface the right internal policy document even if your question used different terminology than the document itself.
A keyword search is like going to a library and asking for books that contain the word "cardiac." A semantic search is like asking the librarian "I need books about heart attacks" — and the librarian brings you books on myocardial infarction, coronary artery disease, and emergency cardiac care, even though none of them used the phrase "heart attack" in their title. Same underlying need, much better results.
You've already used RAG. Every time you use an AI with web search enabled — ChatGPT with browsing, Claude with web search, Perplexity — you're using the simplest version of this pattern.
Your question goes in. Before the model answers, a search query goes out to the web. The top results come back. Those results get injected into the context window alongside your question. The model reads the search results and your question together, then generates a response that's grounded in current web content rather than training data alone.
The model didn't get smarter or more current. The architecture around it retrieved current information and handed it over. The result feels like the model "knows" recent news — but really it just read it the same way it reads everything else you put in the context window.
Web search handles the time gap. For private knowledge — your internal documentation, your customer data, your proprietary processes — you build a private knowledge base instead.
The pattern is the same: your documents get indexed and vectorized in advance. When a question comes in, the retrieval system finds the most relevant chunks. Those chunks go into the context window. The model answers from them.
This is how organizations deploy AI assistants that can answer questions about their own products, policies, and systems — without sending any of that private data to be baked into a shared model's weights. The knowledge stays in your retrieval system. The model only sees it at the moment it's needed, within the context window of that specific conversation.
This distinction matters a lot. Training a model on your data means that data is encoded in the model's weights — potentially surfaceable to other users in the right conditions, and difficult to remove. RAG keeps your data in your retrieval system, under your control. The model sees a document for one conversation. It doesn't retain it. Next conversation, clean slate.
RAG isn't magic. A few limitations worth understanding:
Retrieval quality determines answer quality. If the retrieval step pulls the wrong documents — because the question was ambiguous, the knowledge base was poorly organized, or the semantic match failed — the model answers from the wrong source. Confidently. It doesn't know the retrieved documents were wrong for the question.
Context window limits apply. You can only inject so much retrieved content before you hit the model's context limit. For very large document sets or very complex questions, you can't retrieve everything relevant — you have to pick. Getting that selection right is an engineering problem, not an AI problem.
The model still fills gaps with training knowledge. If the retrieved documents don't fully answer the question, the model doesn't say "I only have partial information." It blends retrieved content with trained knowledge — sometimes seamlessly, sometimes in ways that introduce the model's own assumptions where the documents were silent.
What knowledge base would make an AI assistant genuinely useful in your context? Not "all company documents" — that's too broad. Think specifically: what are the three or four sources that, if the model could read them before answering, would make it dramatically more useful for the questions people actually ask? That specificity is where RAG implementations succeed or fail.