RAG that cites its source and says when it does not know
Answer
A knowledge assistant earns trust with two behaviors no model provides by default: every answer carries a citation that resolves to the exact passage it rests on, and a question the corpus cannot answer gets a plain refusal instead of a fluent guess. Both are design choices in the retrieval system around the model, and both can be tested before anyone relies on the answers. The failure modes that break these systems are mostly retrieval failures, not model failures, which is why a system nobody measures drifts from citing to guessing without anyone noticing.
TL;DR
- The two behaviors that make a knowledge assistant trustworthy, citing the exact source passage and refusing when the answer is absent, are engineering choices, not model features.
- Most failures are retrieval failures: the answer was not found, was outranked, or was buried mid-context where models measurably lose it. The model then answers from its own memory, fluently.
- Citation-or-abstain is a hard rule, not a style: no supporting passage, no answer. The refusal is a first-class result that tells you what to fix.
- Measure retrieval separately from answers: an engineering study of RAG failures concluded that validation is only feasible during operation, so the measurement has to be built in, not run once.
- Some corpora should not get a RAG system at all: structured values belong in a database query, and a corpus nobody owns automates the distribution of stale answers.
What makes a knowledge assistant trustworthy?
Two behaviors make a knowledge assistant trustworthy, and neither comes from the model. The first is that every answer carries a citation, and the citation resolves to the exact passage the answer rests on, not to a document title the reader must go excavate. The second is that a question the corpus cannot answer gets a plain refusal, in words, instead of a fluent guess. Retrieval-augmented generation, the technique behind these assistants, promises grounded answers; whether a given system keeps that promise is decided by the engineering around the model, and both behaviors can be tested before anyone relies on the output.
The alternative is familiar to anyone who has piloted one of these systems: an assistant that answers everything, beautifully, and is sometimes wrong in ways nobody can trace. That assistant does not fail loudly. It substitutes its own training memory for your documents the moment retrieval comes back thin, and the substitution reads exactly like a grounded answer, minus the part where it is true.
Where do these systems actually fail?
They fail mostly in retrieval, before the model writes a word. An engineering study of RAG systems built across three separate domains, research, education, and biomedical, catalogued seven distinct failure points and drew a conclusion worth taking seriously: validation of such a system is only feasible during operation, and its robustness evolves rather than being designed in at the start.01 The failure modes we design against fall into a short list:
| Failure | What the user sees | What the mechanism does |
|---|---|---|
| The answer is in the corpus, but retrieval never fetched it | A confident answer from the model's own memory | No supporting passage, so the answer is refused; the refusal is logged against the query |
| The right passage was fetched, then buried mid-context | A partial or drifted answer | Fewer, better passages ranked deliberately; position is treated as a design variable |
| The passage was fetched, and the answer contradicts it | An answer the citation does not support | A support check compares answer to passage before anything ships; mismatch becomes a refusal |
| The answer is not in the corpus at all | A guess, dressed as an answer | "Not in the corpus" returns as a first-class result, and the gap is a work item |
| The answer is in a document this user may not read | A leak | Retrieval runs behind the same permissions the documents already have |
The failure modes a citation-or-abstain design converts from silent wrongness into visible, fixable events.
The mid-context failure deserves its own sentence, because it is measured rather than anecdotal: language models use long inputs unevenly, performing best when the relevant information sits at the beginning or end of the context and significantly worse when it sits in the middle, even for models built for long contexts.02 Stuffing more passages into the prompt is therefore not a safety margin. It is a way to hide the right passage where the model is measurably worst at finding it.
What does citation-or-abstain mean in practice?
Citation-or-abstain means the system is not allowed to answer without evidence, as a rule enforced outside the model. An answer ships only when a retrieved passage is attached to it, and a check confirms the passage actually supports what the answer says. Fail either condition and the user gets the honest sentence instead: the answer is not in the corpus. The refusal is not an error state to be minimized in a demo; it is a result. Each one is logged with its query, and the log becomes the maintenance plan: a cluster of refusals names a document that should be in the corpus, or a retrieval gap in front of one that already is.
This is also why the citation must resolve to the passage, not the document. A citation to a two-hundred-page PDF is theater; nobody checks it, which means it disciplines nothing. A citation to the paragraph invites checking, and a system whose answers get checked is a system whose failures get found while they are cheap.
How do you know it works, and keeps working?
You know it works by measuring retrieval separately from answers, on a graded set of real questions, and by keeping the measurement running after go-live. The people who will use the assistant write the questions, including questions whose correct answer is "not in the corpus", because an assistant that cannot refuse has not been tested on the case that breaks trust. Retrieval is scored on its own: was the passage that supports the correct answer in what was fetched? Then the answers are scored against the same set. Two numbers, because they fail independently and are fixed by different work.
The study above found that validation is only feasible in operation,01 and our run discipline follows from that: the question set is versioned, refusals and corrections feed it, and the scores are watched like any other production metric. A knowledge assistant is not finished at go-live; the corpus drifts, the questions drift, and what to evaluate, offline and online is standing work, not a launch task. Who signs the bar those scores are measured against, and on whose documents, is the subject of is 95% accuracy good enough.
When should you not build this?
Three cases fail the fit test, and naming them is part of the design. When the answers are exact values living in structured systems, invoice totals, policy numbers, account balances, a database query is the correct tool, and retrieval over documents is a slower, fuzzier way to be wrong about facts you already store precisely. When the corpus is small and stable, a handbook, a policy pack, the honest comparison is against putting the whole thing in front of the model directly; retrieval earns its complexity only when the corpus outgrows that. And when nobody owns the corpus, no one updates it, no one retires stale documents, the assistant does not fix your documentation problem. It automates the distribution of stale answers, now with citations attached.
The assistants we build are scoped to survive these questions: grounded in the documents you actually maintain, behind the permissions you already run, with the refusal designed in and the measurement running. If your pilot answers everything and cites nothing you can check, that is not a model problem, and a better model will not fix it. The free readiness check scores a workflow against the seven production gates in three minutes; a workflow review is thirty minutes on your actual corpus and questions.
Sources
Every load-bearing claim above, with its source and the date we checked it.
| Source | Reference | Accessed |
|---|---|---|
| 01 Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System (arXiv 2401.05856), basis verified | https://arxiv.org/abs/2401.05856 (opens in a new tab) | |
| 02 Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172), basis verified | https://arxiv.org/abs/2307.03172 (opens in a new tab) |
Frequently asked questions
- Why does a RAG assistant give wrong answers confidently?
- Because when retrieval fails, the model does not stop; it answers from its own training memory in the same fluent register. The wrongness is upstream of the words: the right passage was never found, was outranked by weaker matches, or sat mid-context where models measurably lose information. A system that must cite or abstain turns that silent substitution into a visible refusal.
- Can the model just say when it does not know?
- Not reliably on its own; abstention has to be designed. The working shape is a rule outside the model: an answer ships only with a supporting passage attached, a checker confirms the passage actually supports the answer, and anything that fails either test returns 'not in the corpus' instead. The refusal is logged as a result, because each one names a gap in the corpus or the retrieval.
- How do you test a knowledge assistant before trusting it?
- With a graded set of real questions from the people who will use it, including questions whose correct answer is 'not in the corpus'. Score retrieval on its own: was the right passage in what was fetched? Then score the answers, and keep both measurements running after go-live, because the corpus and the questions drift.
- When is RAG the wrong tool?
- When the answers are exact values that live in structured systems, a database query is correct and a retrieval system is a slower, fuzzier way to be wrong. When the corpus is small and stable, putting it directly in front of the model beats building retrieval. And when nobody owns the corpus, the assistant automates the distribution of stale answers with citations attached.