In RAG the problem isn't the answer, it's the document
- RAG
- Production
The complaint always arrives in the same sentence: "the model is making things up".
Look closely and most of the time it is not. It is summarising the document it was handed, quite accurately. The document is wrong.
That distinction is half the work, because the two problems live in different places. A model problem gets solved with prompts and model choice. A retrieval problem gets solved in the index, the chunking and the query.
What recall@k doesn't show you
The metric people reach for first is recall@k: is the correct document in the top k.
It measures something real. It misses three things that matter in production.
The right document arrived, but at what rank? A correct document at position five sits behind a wrong one at position one inside the context. How many wrong documents came along with it? If six of seven are irrelevant, the model splits its attention. And most importantly: what happens when the answer is in no document at all? Can the system say "I don't know", or does it manufacture an answer from whatever was closest.
The last one does the most damage. In a corporate knowledge base, questions wander out of scope constantly, and the right behaviour there is to produce nothing.
You can't fix what you can't see
The real obstacle to improving retrieval was a lack of visibility. Answers come back bad, nobody knows which documents produced them, and the discussion runs on guesswork.
Our simplest and most useful change was storing the retrieved document ids and scores alongside every answer. A day's work that ended an argument we had been having for months.
After that, bad answers sorted into two buckets. Right document arrived and the answer is still bad: the problem is the model or the prompt - it was there and the model ignored it or misread it. Right document never arrived: the problem is retrieval, and the model did what it could with what it had.
Every improvement attempt made before that split was running blind.
Chunking comes before the embedding model
When retrieval is poor the first instinct is to swap the embedding model. For us its effect was small next to fixing chunking.
Fixed-length chunking cuts straight through semantic boundaries. A chunk sliced out of the middle of a procedure contains neither the answer nor its context when read alone.
We anchored chunking to the document's own structure instead: heading hierarchy, list boundaries, table integrity. Chunks are not uniform in size but each one reads on its own.
The second gain came from somewhere we did not expect. Attaching each chunk's heading chain to it - keeping "Backups > Weekly schedule > Exceptions" inside the chunk - noticeably improved both retrieval and the model side.
Filter before the search, not after
Corporate questions are almost always scoped: a specific environment, a specific customer, a specific date range.
Applying those constraints after the search goes badly. The entire top-k can fall out of scope and leave you with nothing - while a perfectly good document sits in the index.
Metadata filters belong in the query, not in a post-hoc pass.
Improving the model side of RAG is visible work. Improving retrieval is invisible. Most bad answers come from there.
The order that worked for us: record what was retrieved, sort bad answers into two buckets, then look at chunking. The embedding model argument comes after all of it.