Use the observed failure to choose the next experiment. Supplying evidence, changing learned behavior, and expanding available context can complement each other, but they solve different problems.
Begin with one failed answer
Imagine a fictional internal assistant that answers questions about an organization's travel policy. It gives an outdated meal allowance and formats the response inconsistently. Those symptoms should be investigated separately.
The first question is whether the current policy reached the model. If it did, inspect whether the answer used the right passage and version. The formatting problem may need a different intervention. Buying a larger context window or starting a training job before examining these cases can hide the actual failure.
RAG changes the evidence supplied at inference
Retrieval-augmented generation combines generation with access to retrieved material. The original RAG work formalizes a model that combines parametric and retrieved non-parametric memory. [S1] In an application, a retrieval stage can supply selected passages from a document collection alongside the user's question.
For the travel assistant, an experiment could retrieve the current policy section and ask for an answer citing that passage. Test retrieval and answer quality separately. If the required section never enters the prompt, improving the answer-writing prompt alone has not repaired the missing evidence.
Updating an index is also different from changing the model weights. That distinction makes the update path inspectable, but it does not automatically make the retrieved content current, authorized, or correct.
Fine-tuning changes learned behavior
Fine-tuning adapts model parameters through training. LoRA is one parameter-efficient approach: it freezes base weights and trains low-rank updates for selected transformations. [S2] This changes how the model behaves rather than serving as a live document lookup.
For the fictional assistant, consistently following a particular response format might be an adaptation target if simpler prompting and validation are insufficient. A useful experiment needs representative training examples and held-out evaluation cases.
Training on a policy document does not create a transparent, continuously updated policy database. If an allowance changes again, the organization still needs a reliable path for supplying current evidence. A formatting improvement should also be checked for regressions in factual answers.
Longer context changes how much evidence can fit
A larger context can make it possible to include more material directly. It does not establish that every relevant detail will be used correctly. Lost in the Middle found position-sensitive performance in the models and tasks it studied; it is a reason to test information placement, not a universal score for every later model. [S3]
For the travel assistant, a direct-context baseline could include a small, versioned policy packet. Compare it with retrieval on the same questions. Measure answer quality, citations, input size, latency, and update effort rather than assuming that the approach with the most supplied text wins.
Swipe or scroll sideways to see every column.
| Observed problem | First experiment to consider | What to check |
|---|---|---|
| Current policy is absent | Retrieve or directly supply the correct passage | Version, coverage, permissions |
| Evidence is present but ignored | Test wording, structure, and passage position | Whether the answer uses the supplied evidence |
| Output format is inconsistent | Try examples and validation before adaptation | Format compliance and factual regressions |
| A stable behavior still needs adaptation | Evaluate supervised or parameter-efficient fine-tuning | Held-out performance and maintenance cost |
Combine approaches only after the tests explain why
These are not mutually exclusive categories. An adapted model can receive retrieved documents, and a retrieval system can use a long context. The question is which added component improves a measured failure enough to justify its cost and maintenance.
I would keep a small versioned test set containing current-policy questions, ambiguous requests, missing-evidence cases, and formatting checks. Change one part of the system, rerun those cases, and inspect the failures. That makes the architecture decision a result of evidence rather than a competition between fashionable labels.