What is RAG and when do you need it?

RAG, or retrieval-augmented generation, is a way of getting AI to answer questions from your own information. Instead of relying on what a model learned in training, the system first searches your documents for the relevant passages, then gives those passages to the model along with the question and asks it to answer from them.

It is the technology behind most internal knowledge assistants, help center bots and document question tools. This guide explains how it works in plain English, when it is the right choice and what separates a good RAG system from a frustrating one.

The problem RAG solves

Language models know a lot about the world, but nothing about your business: your policies, products, prices, procedures or customer history. Asked about them, a model either says it does not know or, worse, invents a plausible answer. RAG fixes this by giving the model the right information at the moment it needs it.

How RAG works, step by step

1. Prepare the documents. Documents are split into passages, often called chunks, each small enough to be useful on its own.
2. Index them. Each chunk is turned into embeddings, numbers that represent its meaning, and stored for search. Many systems also index keywords.
3. Search. When a question arrives, the system finds the chunks closest in meaning, and often those matching key terms.
4. Rerank. A second step orders the results more precisely.
5. Answer. The best chunks go to the model with instructions to answer only from them and cite sources.

When you need RAG

• Answers must come from your own documents, not general knowledge.
• The information changes: prices, policies, procedures, product details.
• People need to see the source to trust the answer.
• Different users may see different documents.
• The collection is too large to send to a model in one go.

Typical uses include staff help desks, customer support assistants, policy and contract search, and technical documentation assistants. See knowledge base search and internal help desk.

When you do not need RAG

• The documents are few and short enough to include directly in each request.
• The task is about style or format rather than facts.
• The information lives in a database better queried directly, such as order status.

For the difference between RAG and training a model on your data, read RAG vs fine-tuning. In short, fine-tuning teaches style and patterns, while RAG supplies facts.

What makes RAG good or bad

Most RAG problems come from the search step, not the model. If the right passage is not found, even the best model cannot answer well. The main quality levers are:

• Chunking. Splitting documents at sensible places, keeping headings with their sections.
• Hybrid search. Combining meaning-based search with keyword search, so product codes and names are found.
• Reranking. Reordering the top results so the best passages come first.
• Metadata. Filtering by document type, date, department or permissions.
• Clean sources. Removing outdated and duplicate documents.

Designing good answers

Instruct the model to answer only from the provided passages, cite them and say clearly when the answer is not in the documents. Keep answers short, with a link to the source for detail. These instructions, together with good retrieval, are the main defense against hallucination.

Permissions and privacy

Not everyone should see every document. A good RAG system filters search results by the user's access, so a sales rep never sees HR files and customers never see internal notes. Decide which AI services may process your documents, and under what terms. See how to keep customer data safe when using AI.

Keeping it current

RAG is only as current as its index. Set up automatic updates when documents change, remove old versions and review the collection regularly. One of RAG's biggest advantages is that updating a document updates the answers immediately, without retraining anything.

What building one involves

1. Choose one collection and one audience to start, such as HR policies for staff.
2. Gather and clean the documents.
3. Collect real questions and correct answers for testing.
4. Set up chunking, indexing and search.
5. Write the answer instructions and test.
6. Add permissions, logging and feedback buttons.
7. Launch to a small group, review questions weekly and improve.

Choosing the parts

A RAG system needs an embedding model, a place to store vectors and a language model to answer. Vector search can live in a dedicated vector database or in an existing database such as PostgreSQL with a vector extension. For models, see our embedding models and how to choose an AI model for your business.

Measuring success

• Share of test questions answered correctly with the right source.
• Share of questions where the assistant correctly declines.
• User feedback on answers.
• Time saved compared with searching manually or asking colleagues.

Beyond simple RAG

Once basic RAG works, many teams add agent abilities: the assistant can search, then look up a record, then draft a document. That is where RAG meets AI agents. Start simple and add capability as trust grows. See what are AI agents and how do businesses use them.

An example

A professional services firm has thousands of past reports, proposals and policies in shared drives. Staff spend time searching and asking senior colleagues. A RAG assistant indexes the approved documents, filters by team permissions and answers questions with links to sources. New staff get answers in seconds, and senior people are interrupted less. This is an illustrative scenario that reflects a common use.

Common pitfalls

• Indexing everything, including outdated and duplicate files.
• Ignoring permissions until late in the project.
• Testing with easy questions only.
• No way for users to flag wrong answers.
• No owner responsible for keeping content current.

Who owns a RAG system

A RAG assistant is only as good as the content behind it, so someone must own that content. Name a person or team responsible for keeping documents current, removing outdated versions and reviewing questions the assistant could not answer. Those unanswered questions are valuable: they show where your documentation has gaps. Many teams find that a RAG project improves their documents as much as it improves access to them.

On the technical side, someone should monitor search quality, costs and errors, and rerun the test questions after any change to documents, chunking or models. A short monthly review is usually enough to keep answers reliable as the collection grows.

Start small, with one collection and one group of users, and let real questions guide what you add next.