Try this. Ask any popular AI model what your company's return policy is. It will give you a confident, nicely worded answer, and it will be wrong, because it has never seen your policy. It filled the gap with what return policies usually say.
That's the core problem with using a general model for business questions. It knows a lot about the world and nothing about you. Retrieval-augmented generation, usually shortened to RAG, is the most common fix.
The idea is simple
Instead of asking the model to answer from memory, you do two steps. First, search your own documents for the passages most likely to contain the answer. Second, hand those passages to the model along with the question and tell it to answer using only that material.
It's the difference between asking a new hire to guess and handing them the handbook open at the right page.
What you actually build
There are three moving parts, and none of them is exotic.
You split your documents into chunks, usually a few paragraphs each. You turn every chunk into a list of numbers (an embedding) that captures its meaning, and store those in a vector database or a search index. When a question arrives, you convert it the same way and pull the closest chunks.
Then comes the prompt: here's the question, here are five passages, answer from these, and say so if the answer isn't in them. That last instruction matters more than people expect.
Where it goes wrong
RAG is not a cure for wrong answers. It moves the failure point, and it's worth knowing where.
Bad search. If the right passage never gets retrieved, the model has nothing good to work with. Most disappointing RAG projects fail here, not in the model. Chunks that are too long dilute the meaning, and chunks that are too short lose context.
Messy source documents. Old PDFs, three versions of the same policy, tables that turn into gibberish when extracted. The model can't tell which version is current unless you tell it. Clean the content before you blame the AI.
Confident guessing. Even with good passages, a model may fill gaps. Telling it to admit when it doesn't know helps, but doesn't remove the problem. Showing the source passage next to the answer lets people check in a few seconds.
Access control. If your index holds HR files and sales pricing together, the assistant may quote either to anyone. Filter by who's asking before the search, not after.
Do you need RAG or something simpler?
If you have a few dozen pages of stable information, you may not need a vector database at all. You can often paste the whole thing into the prompt. If your content changes weekly and runs to thousands of pages, RAG makes sense. Fine-tuning a model on your content is rarely the right first move for facts, since it's expensive and the knowledge goes stale the moment a policy changes.
A sensible way to start
Pick one source, such as your help centre or product manual. Write twenty real questions that customers or staff ask, along with the correct answers. Build the simplest version, run those twenty questions, and read every answer. You'll learn more from that hour than from any vendor demo.
If it works, grow the content. If it doesn't, the failures will tell you whether the problem is search, documents or the prompt.
