Citations or it didn't happen: RAG that shows its sources

Most RAG demos answer confidently and cite nothing. That is fine for a weekend prototype and a liability in production. The moment a support agent or a customer reads an answer, the first question is "where did that come from?" If the system cannot answer that, people stop trusting it.

This is the shape I use when citations are a hard requirement.

#Chunk for citation, not just for recall

The usual advice is to split documents into fixed windows of a few hundred tokens. That works for recall, but it produces chunks that start mid-sentence and cannot be quoted cleanly.

Instead, split on structure first, then size:

• Split on headings and paragraphs so each chunk is a coherent passage.
• Merge small neighbours until a chunk reaches the target size.
• Keep a short overlap so an idea that spans a boundary is not lost.
• Store the source id, heading path and character offsets with every chunk.

Those offsets are what make a citation clickable later.

type Chunk = {
  id: string;
  docId: string;
  headingPath: string[]; // ["Billing", "Refunds"]
  start: number;         // character offset in the source
  end: number;
  text: string;
  embedding: number[];
};

#Retrieve more than you show

Vector search alone misses exact terms like product codes and error messages. I run a hybrid query: vector similarity plus a keyword score, then re-rank the combined top results.

With Postgres and pgvector that can stay in one query:

select id, doc_id, text,
       1 - (embedding <=> $1) as semantic,
       ts_rank(to_tsvector('english', text), plainto_tsquery($2)) as keyword
from chunks
order by (0.7 * (1 - (embedding <=> $1))
        + 0.3 * ts_rank(to_tsvector('english', text), plainto_tsquery($2))) desc
limit 20;

Retrieve around 20, re-rank, and pass the best 5 to 8 to the model. The extra candidates cost little and give the re-ranker room to fix what the embedding missed.

#Make the model cite by construction

Asking a model to "please cite sources" is not enough. Give every passage a short, stable label and require those labels in the answer:

Answer using only the passages below. After every sentence that uses a
passage, add its label in square brackets, like [P2]. If the passages do
not contain the answer, say you don't know.

[P1] (Billing > Refunds) Refunds are issued to the original payment method...
[P2] (Billing > Refunds) Annual plans can be refunded within 14 days...

Then validate the output. Any label that was not in the prompt is a hallucinated citation, and the answer should be retried or flagged.

#Render citations people can check

On the client, turn each [P2] into a small footnote chip. Hovering or tapping it shows the passage, its heading path and a link that opens the source scrolled to the stored offsets. When readers can verify one answer in a click, they stop feeling the need to verify all of them.

#Keep a human in the loop

For support workflows, I keep a review queue: answers with low retrieval scores, no citations, or a "don't know" go to a person first. Their edits become new evaluation cases, and the evaluation set is what tells you whether the next change to chunking or prompting actually helped.

Citations are not a UI flourish. They are the contract between the system and the people who rely on it.