Loading lesson…
Applied LLM Systems: RAG, Agents & MCP / Retrieval-Augmented Generation
From documents to an assembled prompt.
Reviewed by Yuvaraj
You now have the pieces: embeddings turn text into vectors, and vector search finds the nearest ones. A RAG pipeline is those pieces in order, with a few more, turning a pile of documents and a question into a grounded prompt. The value of seeing it as a pipeline is that every stage is inspectable, when an answer is wrong, you can find out which stage failed.
Answer from memory before revealing, retrieval practice is what builds durable recall.
In a RAG pipeline, what is sent to the language model at generation time?
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Chunk too large and each passage mixes several topics, so retrieval is imprecise and you waste context. Chunk too small and you sever the sentence that carried the answer. Overlap between chunks hedges against splitting a key idea across a boundary. There is no universal best size, it depends on the documents.
A quick count makes the knobs concrete: a 1,000-word document, split into 200-word chunks with a 20% (40-word) overlap, advances 160 words per chunk, so it yields about 6 chunks, each sharing a little text with its neighbour. Halve the chunk size and you double the chunk count (and the index size); drop the overlap and you save space but risk cutting an answer in two.
For a curious beginner
Imagine turning a textbook into index cards. Cards that are too big cover several ideas at once, so the "most relevant card" is only partly relevant. Cards that are too small cut a thought in half. Chunking is choosing a card size, and letting cards overlap a little so no sentence falls through the crack.
How it is actually used
Chunk by a fixed window of words or tokens with a set overlap, or split on structure (headings, paragraphs). Each chunk is embedded once and stored with its source id so the answer can cite it. Retrieval returns the top-k by cosine; a reranker may reorder them. The assembly step concatenates the kept chunks under a fixed instruction template, respecting a character or token budget so the prompt fits the context window.
The underlying mechanism
Retrieval scores each chunk vector against the query vector by cosine similarity and keeps the top :
The assembled context is the ordered concatenation of those chunks under a token budget : chunks are added by descending until the budget is reached.
This is the point learners most often miss: the model does not search anything. By generation time, retrieval is already done, and the model sees one ordinary prompt, your instructions, the retrieved passages, and the question. Its answer is only as good as the passages the earlier stages selected.
This lab does not fabricate an answer
The linked RAG Pipeline Simulator runs every stage for real, chunking, embedding, retrieval, and context assembly, and then stops at the prompt. The generation panel shows the exact prompt that would be sent to a model, with an explicit notice that connecting a real model is what produces an answer. It never invents model output, because a fake answer would teach the wrong thing.