Explainer

RAG makes more sense when you start with a handbook

A clear example of retrieval-augmented generation, including where retrieval fails and when a system should say it cannot answer.

By Saud AlotaibiPublished 3 October 20263 min read
The takeaway

Check whether the right evidence was retrieved before trying to improve the generated answer.

Imagine a company handbook with a policy for borrowing equipment. You want a chat tool to answer, “Can I take a monitor home for the weekend?” The question is ordinary language. The policy might use the phrase “temporary off-site equipment loan.” A simple word match may miss the connection.

A retrieval-augmented generation system, usually shortened to RAG, searches a collection for relevant material and supplies that material to a model when it generates an answer. In this example, the system should find the equipment policy before composing a response.

Separate the two jobs

Retrieval chooses evidence. Generation turns the question and evidence into an answer. Those jobs fail in different ways, so evaluate them separately.

Suppose the search retrieves an office furniture purchasing policy. The language model may produce an elegant answer, but it is working from the wrong page. Changing the tone instruction will not fix that. You need to investigate document preparation, the search query, or the ranking of results.

Now suppose the correct paragraph is retrieved, but the answer drops the manager-approval requirement. That is an answer-quality problem. The evidence existed, and the response failed to preserve an important condition.

Preserve context when splitting documents

A document collection is often divided into smaller pieces so a search system can retrieve relevant passages. For this handbook, keep the heading, policy text, and nearby exception together when possible. A fragment containing only “requires approval” is hard to interpret without knowing what requires approval.

Keep a source identifier and section title alongside each passage. Those fields help a reader verify the answer and help the developer investigate a bad retrieval. Avoid stripping all structure merely to obtain clean-looking plain text.

Before adding complicated chunking rules, inspect a few real passages. Can a reader understand each one by itself? Are tables broken across fragments? Do page headers overwhelm the policy? Those questions are often more useful than choosing a chunk size by habit.

Build a question set before the demo

Write several realistic questions. Include a straightforward policy question, a paraphrase, an exception, and a question the handbook cannot answer. For each answerable question, record the supporting passage.

An illustrative test set might include:

  • Who approves a temporary equipment loan?
  • Does the policy cover contractors?
  • What happens if borrowed equipment is damaged?
  • Does the company reimburse home internet?

The last question may have no answer in this handbook. A useful system should communicate that gap rather than invent a policy.

Use citations as a reading aid

A source link helps only if it points to material that supports the nearby claim. Ask the system to preserve conditions and distinguish a direct policy statement from an inference. A citation should make checking faster, not decorate a response.

Semantic search can help connect differently worded questions and passages, but similarity does not establish truth or permission. If two policy versions disagree, the system needs a rule for which version is current. That rule belongs in document management.

For a small first build, start with one reliable handbook, a clear version date, and a modest test set. Make retrieval visible while developing. A system that admits “I could not find this in the handbook” is more useful than one that fills every gap with confident prose.

Sources & further reading