Many businesses want an AI assistant that answers questions about their own products, policies and documents. The obvious obstacle is that a general-purpose language model does not know your documents. It was trained on a large body of public text, and it stops learning on a training cutoff date.
Retrieval-augmented generation, usually shortened to RAG, is the most common way around that. This article explains what it is, what it is good for, and where it falls short, in plain terms.
What RAG is
A large language model (LLM) generates text from patterns it learned during training. Left alone, it can only draw on that stored knowledge, and it can state wrong things with confidence.
RAG adds a step before the model answers. The system first searches a collection of your documents for the passages most relevant to the question. It then hands those passages to the model along with the question, and the model writes its answer from them.
The term comes from a 2020 research paper by Patrick Lewis and colleagues, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." They paired a text generator with a searchable index of Wikipedia passages, so the output could draw on retrieved knowledge instead of only what the model stored in its parameters.
OpenAI's retrieval documentation describes the same pattern in practical terms: your files are split into chunks, converted into numerical representations called embeddings, and indexed. At question time the system retrieves the relevant chunks and passes them to a model with the original query, and the model writes its answer from those sources.
Think of a capable new hire who has never seen your manuals. RAG does not train that person. It puts the right page in front of them before they answer.
Why businesses use it
Answers come from your material
The model's answer is built from passages in your own documents, so it can reflect your current policies, prices and procedures. When a document changes, you update the document and the index, not the model.
You can show the source
A well-built RAG system can return the passage or document it used. That lets a person check the answer instead of taking it on faith. It is the single most useful property for any use where being wrong costs money.
No model training required
RAG works with an existing model. Adding new knowledge means adding documents, which is faster and cheaper than training a model.
What RAG does not fix
RAG reduces one cause of wrong answers, which is the model not having the information. It does not eliminate wrong answers.
- Retrieval can miss. If the system fetches the wrong passages, the model answers from the wrong material. Search quality decides a lot.
- Bad documents produce bad answers. Outdated price sheets, conflicting policies and scanned PDFs with poor text all flow through.
- The model can still misread a good passage. Checking answers against sources is still necessary.
- It adds access questions. If the index includes files that not everyone should see, the assistant may reveal them. Permissions have to carry through to the search.
RAG compared with just pasting documents into a prompt
Modern models accept long inputs, so a fair question is why you would build retrieval at all. For a short policy or a single price sheet, you often should not. Paste it in and test the answers. Retrieval earns its keep when the material is too large to include every time, when different questions need different documents, or when the content changes often enough that a pasted copy goes stale. The cost of retrieval is moving parts: a document pipeline, an index to keep current, and a search step that can fail on its own. Add those only when the simpler version stops working.
Where it fits in a service business
Everything below is an illustration, not a result from a specific company.
- Technician support: a tech on a job asks about a model number's error code, and the system pulls the relevant page from the manufacturer's manual and your own service notes.
- Customer service: a customer asks whether a repair is covered under their maintenance agreement, and the assistant answers from the agreement terms and cites the clause.
- Internal policy: a new CSR asks how your company handles after-hours emergencies, and the assistant answers from your written procedure.
- Sales support: a comfort advisor wants the warranty terms for a specific equipment line before an estimate.
For a trades company, the value is rarely a flashy chatbot. It is getting the right paragraph in front of the right person in seconds. If you are working out whether an assistant like this belongs on your list of projects, Di-Hy's AI strategy consulting can help you decide whether a project like this is worth doing. Our guide to AI agents for service businesses covers how retrieval fits with tools that take actions, and our AI compliance guide for small service businesses covers the data-handling questions to ask before you index customer files.
How to start small
- Pick one narrow question set. For example, warranty terms or one product line, not the whole company.
- Clean the documents first. Remove outdated versions and decide which file is the source of truth.
- Test with real questions. Write 30 to 50 questions people actually ask, with the correct answers, and check how often the system gets them right.
- Require citations. Make the assistant show where an answer came from, and have a person review a sample each week.
- Control access. Index only what everyone using the assistant may see.
If it works on a narrow set, widen it. If it does not, fix the documents or the search before adding more.
Frequently asked questions
Is RAG the same as fine-tuning?
No. RAG supplies relevant documents to the model at the moment of the question. Fine-tuning changes the model itself by training it on examples. Our explainer on supervised fine-tuning covers the other approach.
Does RAG stop AI from making things up?
It reduces the problem by giving the model source material to work from, but it does not eliminate it. Retrieval can miss, and the model can still misread a passage. Requiring citations and sampling answers for review are the practical safeguards.
Do I need RAG for a small business?
Not always. If the information fits in a single prompt, such as one price sheet or a short policy, you may not need a retrieval system. RAG becomes useful when the material is too large or changes too often to paste in.