Retrieval and fine-tuning get discussed as two routes to the same destination. They do different jobs, and the difference is easy to state. Retrieval gives a model facts it did not have. Fine-tuning changes how a model behaves and leaves what it knows roughly where it was. Teams that mix those up buy the expensive option to solve a problem it was never going to solve.
Here is a way to decide that holds up once the work starts.
Retrieval, in plain terms
Retrieval means that before the model answers, your system searches your own material, pulls back the passages that look relevant, and puts them into the prompt. The model then answers using text it can see in front of it. Nothing about the model changed. You changed what it was looking at.
That is why retrieval is the right tool whenever the correct answer depends on information that moves. Prices, policies, patient records, stock levels, last week's release notes. Update the source and the next answer is correct, with no retraining and no deployment.
Fine-tuning, in plain terms
Fine-tuning means taking a model and continuing its training on examples of the behaviour you want. Given this kind of input, produce this kind of output. Do that a few thousand times and the model gets reliably better at that shape of task.
It is good at format, tone, house style, and performing one narrow task the same way every time. It is good at pulling fields into exactly the schema your database expects. It is bad at memorising facts. You can fine-tune a model on your product documentation and it will start sounding like your documentation while still inventing details. The style transfers. The accuracy does not.
Retrieval changes what the model can see. Fine-tuning changes how it speaks. If the problem is that it makes things up, only one of those helps.
The rule
If the answer must reflect data that changes, retrieve. If the answer must arrive in a consistent shape, or the task is narrow and repetitive and the general-purpose behaviour keeps drifting, fine-tune. If both are true, do both, and build the retrieval first.
The order matters more than the choice. Retrieval is cheaper to build and faster to change, and it tells you inside a week whether the idea works at all. Fine-tuning bakes your assumptions into a set of weights you then have to maintain. Start with the reversible option. A lot of teams find the retrieval version is good enough and never come back for the other one.
Most retrieval failures are search failures
When a retrieval system gives bad answers, teams reach for a better model. That almost never helps, because if the right passage was never retrieved, no model can answer from it. Check that first, and check it directly.
The concrete test: take twenty questions you already know the answers to, run only the retrieval step, and read what comes back. Ignore the generated answer entirely. Did the passage containing the answer appear in the results? If it failed to appear for six of the twenty, your ceiling is seventy percent and no amount of prompt work moves it.
What does move that number is unglamorous. How you split documents matters more than almost anything else in the system. Cut a contract every five hundred characters and you will slice clauses in half, so the retrieved passage carries the condition without the exception that followed it. Split on structure instead. Sections, clauses, headings, whatever the document already uses to organise itself.
Then check whether pure vector search suits your data at all. Embeddings are good at meaning and poor at exact strings. If your users search by part number, error code, or surname, run a keyword index alongside and combine the results. A surprising share of retrieval problems come down to a missing plain text index.
When the answer is neither
Before you build either one, rule out the two cheaper fixes.
- The prompt is vague. If your instructions never say what to do when the information is missing, the model will guess. Tell it what to do.
- The scope is too wide. A system meant to answer anything about your company will be mediocre at all of it. One that answers billing questions can be genuinely good.
- The source material is bad. Retrieval over out-of-date documents returns out-of-date answers, faster. That is a content problem wearing an engineering costume.
What you are signing up to maintain
Both options carry a running cost that rarely makes it into the plan.
Retrieval needs its index kept in step with the truth. When a document changes, something has to notice and re-index it. When a document is deleted, the old copy has to leave the index too, or the system will keep quoting a policy you retired last year. Permissions are the part people forget entirely. If the underlying documents have access rules, the retrieval layer has to honour them, or you have built a search engine that leaks.
Fine-tuning needs a dataset kept alive. Change your output format and every training example is stale, while the model carries on producing the old shape. Move to a newer base model and you retrain and re-evaluate from scratch. Budget for that as a recurring cost, not a project with an end date.
The honest summary is that most teams asking this question need better retrieval and a narrower scope, and would get more out of a week spent on their documents than a month spent on training. Fine-tuning is real and sometimes exactly right. It sits further down the list than the conversation around it suggests.