Retrieval-augmented generation demos beautifully. Point a model at a folder of PDFs, ask it a question, watch it answer with a citation. That afternoon’s work is genuinely impressive, and it is also roughly ten percent of a system anyone should put in front of customers.
The other ninety percent is the part nobody films.
The demo lies about your data
Demo corpora are clean. Real ones are a decade of overlapping policy documents, three contradictory versions of the same refund rule, scanned faxes, and a spreadsheet that someone insists is the source of truth.
Retrieval does not resolve contradictions. It surfaces them, and the model picks one — usually the one that happens to embed closest to the question, which has nothing to do with which one is current. Before writing a pipeline, find out which documents are authoritative and which are archaeology. This is a content problem wearing an engineering costume, and it is almost always the longest part of the project.
Build the evaluation set before the pipeline
This is the single highest-leverage thing you can do, and it is the step most often skipped.
Collect fifty to two hundred real questions — from support tickets, sales calls, internal chat — and write down the correct answer and the document it should come from. That is your evaluation set. It takes a day or two of unglamorous work and it changes the character of the whole project, because now every decision has a number attached.
Without it you are tuning by vibes. Someone changes the chunk size, the three questions they happen to test look better, and the change ships. Nobody notices that it got worse for the other forty-seven.
Measure two things separately:
- Retrieval accuracy — did the right passage come back at all?
- Answer accuracy — given the right passage, did the model use it correctly?
Keeping them apart matters, because the fixes are completely different. Bad retrieval is a chunking, embedding, or indexing problem. Bad answers on good retrieval is a prompting or model problem. Teams that measure only the final answer spend weeks tuning prompts to fix what was a retrieval bug.
Chunking is where most of the accuracy lives
The default advice — split every thousand characters with a small overlap — is a reasonable place to start and a bad place to stop.
Fixed-size chunks cut tables in half, orphan a heading from the paragraph it governs, and strip the context that makes a passage interpretable. A chunk reading “this does not apply to enterprise customers” is worse than useless without the section it belongs to.
What tends to work better: split on document structure rather than character count, keep headings attached to their content, and store a little surrounding context alongside each chunk so the model can see where the passage sits. None of this is exotic. It is just more work than calling a text splitter, and it moves retrieval accuracy more than swapping embedding models will.
Make “I don’t know” the easy answer
A model that confidently invents a refund policy is worse than no system at all, because it destroys the trust that made anyone adopt it.
Two things help. First, put a floor on retrieval confidence — if nothing came back above it, say so rather than answering from parametric memory. Second, make abstention explicitly acceptable in the prompt, and include examples of it. Models are agreeable by default; if you never show them a refusal, they will strain to answer everything.
Then measure abstention as a first-class metric. Both directions are failures: answering when it should not, and refusing when the answer was right there.
Citations are a product feature, not a checkbox
Returning the source passage does two jobs. It lets the user verify the answer, and it gives you a debugging trail when something goes wrong — you can see whether the model had the right passage and misread it, or never had it at all.
Cite the passage, not just the document. “This came from the employee handbook” is not a citation when the handbook is ninety pages.
It degrades quietly after launch
This is the part that separates a project from a system. Your accuracy on launch day is the best it will ever be without maintenance:
- Documents get updated and the index goes stale
- Providers change model behaviour under a version alias
- The questions people ask drift as they learn what the system is good at
- Somebody adds two hundred new documents and nobody re-runs the evaluation
Re-run your evaluation set on a schedule and on every change. Track accuracy, cost per query, and abstention rate over time. Alert on regressions. A retrieval system nobody is watching stops being correct long before anyone files a complaint — users just quietly go back to asking a colleague.
The honest summary
RAG is not hard in the sense of requiring novel research. It is hard in the sense that a dozen unglamorous decisions each cost you a few points of accuracy, and the difference between a demo and a system is having measured all of them.
If you are evaluating someone to build this for you, ask how they will measure accuracy and what happens to that number six months in. The answer tells you most of what you need to know.