A language model has broad general knowledge, but does not know a specific company’s current procedures. RAG (retrieval-augmented generation) bridges that gap in two steps: for each question, the system finds passages that answer it in a knowledge base, then gives them to the model with instructions to answer from them.
Where RAG comes from
The method was described in 2020 by Patrick Lewis and his co-authors in the paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, presented at the NeurIPS conference. It has become one of the most widespread approaches to making a model answer from business documents.
How it works, step by step
- Preparation. Documents are split into passages, and each passage receives a representation of its meaning
- Retrieval. For each question, the system finds the passages closest in meaning, even without matching words
- Generation. The model receives the question and these passages, with instructions to answer from them and say when they are insufficient
- Citation. The passages used are displayed with the answer
What it changes in practice
- Knowledge is updated without retraining. A document is edited, and the answer changes as soon as the system has taken the update into account
- Every answer can be checked. When the system displays its sources, source citation shows where the answer comes from
- The model invents less. It sticks to the supplied passages, which limits hallucinations
The limits
RAG is only as good as its documents: an ambiguous passage produces an ambiguous answer, two conflicting versions an unreliable one. Retrieval can also miss the right passage if the document is poorly split or the question uses very different vocabulary. This is why preparing the base and reviewing unanswered questions matter.
RAG or model training?
Fine-tuning a model on your documents requires prepared data and computing time, and must be repeated when information changes. RAG generally leaves the model unchanged and supplies information when needed. For business knowledge that changes often, it is usually the simpler approach, and the two techniques can also be combined.
KernChat assistants work on this principle: their answers come from the organization’s knowledge spaces and cite their source (see how knowledge is prepared).