Fine-Tuning vs RAG vs Prompt Engineering: When to Use Which
Your model keeps answering wrong, and the room splits three ways: "write better prompts," "add RAG," "fine-tune it." Which one is the right answer? Here's the part most comparison articles won't say plainly: usually they're all partially right, because they fix different things. Prompt engineering, RAG, and fine-tuning aren't three competing solutions to one problem — they're three escalating layers, and picking well is less about memorizing feature tables and more about answering one question: are you trying to change what the model knows, or how it behaves?
This article gives you that ruler, a quick three-question filter, and the honest cases where the popular advice falls short.
The upgrade ladder: cheap layers first
Think of LLM customization like onboarding a new hire.
Prompt engineering is the job description. You write clearer instructions — tone, format, constraints — and a capable hire follows them on day one. Cost: nearly zero, since you're only editing text. If the instructions work, you're done; stop reading vendor pages.
Few-shot examples are showing them a couple of worked samples. You paste two or three input-output pairs into the prompt so the model imitates the pattern. Still cheap, still reversible. Meta's own guidance is blunt on the ordering: experiment with in-context learning before any fine-tuning — it even predicts how much fine-tuning would help.
RAG is handing them a reference library. The model answers from retrieved documents instead of memory. Now facts can update daily and answers can cite sources. This is where costs start (vector databases, ingestion pipelines), and we have a full walkthrough of RAG if you're headed there.
Fine-tuning is sending them to formal training. The model's weights are permanently re-shaped by examples you curated. Strongest effect, highest cost, longest lead time — and the least reversible.
(There's a fifth path people forget: long context windows — just pasting everything in. It works for one long document, but it's priced per token, and no window holds a million-document library.)
The rule I actually use in consulting conversations: always exhaust the cheaper layer before paying for the next one. Not because expensive is bad — because if you skip layers, you can't tell whether the expensive one is doing anything. A team that fine-tunes on day one has no baseline to compare against. I've watched teams jump straight to fine-tuning when a paragraph of prompt would have fixed the output format; the training run just taught the model what the paragraph could have said.
One ruler: behavior or knowledge
Databricks frames the core trade-off cleanly: RAG injects knowledge at inference time, while fine-tuning bakes expertise into the weights before deployment. In practice that resolves into a surprisingly reliable split:
- Knowledge problems — the model doesn't know your product manual, this quarter's policy, your contracts — point to RAG. IBM is explicit that fine-tuning is typically not helpful for injecting new knowledge, and a fine-tuned model's knowledge is frozen at training time. New quarter, new regulations? Retrain, or just... update the retrieval index.
- Behavior problems — wrong tone, wrong format, sloppy terminology, ignores edge-case instructions — point to fine-tuning. You're teaching a style or a skill, not a fact.
Prompt engineering sits underneath both: it's the cheap first attempt at either. IBM's guide has a line worth pinning to your wall: further training and greater data access cannot compensate for poor prompting. If your instructions are vague, fixing them costs an afternoon; skipping that step means you'll never know whether RAG or fine-tuning was actually needed.
Prompt engineering: when you hit the ceiling
How do you know prompts have taken you as far as they go? Three signals:
- The model lacks facts, not instructions. No phrasing of the prompt can make it know your private documentation — that's a knowledge gap, and more adjectives won't close it.
- The format won't stabilize. You want strict JSON every time; it complies 90% of the time and freestyles the rest. Instructions get you far; making behavior consistent under load is fine-tuning's home turf.
- The examples outgrew the prompt. When you're pasting a dozen few-shot samples and the model starts ignoring half of them (Meta flags exactly this), you've outgrown the prompt — and, interestingly, that's also the moment RAG enters: retrieving the relevant examples per question is retrieval by another name.
RAG: the knowledge lane
If the diagnosis is knowledge, RAG is the standard answer: connect the model to your documents, retrieve at question time, answer with citations. The mechanism deserves its own article — we've written it — so here's only the decision-relevant part: RAG shines when facts change, sources must be cited, or private data should stay in a controlled layer you can permission and audit.
When is RAG not the fix? When the complaint is "it answers correctly but in the wrong voice/shape/patience." Retrieval cannot teach a model your house style. That's behavior, and the ladder says where behavior lives.
One more honest note: hybrid systems where the model was fine-tuned to use retrieved context better are a recognized pattern — Meta lists "fine-tune an LLM to better use the context from a given retriever" among its example use cases. The lanes help you start; they're not walls.
Fine-tuning: the behavior lane
Fine-tuning continues training a pretrained model on labeled examples — input-output pairs that demonstrate the behavior you want — permanently updating its weights. It's supervised learning: your data quality is the ceiling. Databricks calls data preparation the most critical step, because bad examples don't just fail quietly; they write their errors directly into the parameters.
Three things worth knowing before you commit:
The cost story has changed. Full fine-tuning updates every parameter and wants serious GPUs. Parameter-efficient methods — LoRA (low-rank adaptation) chief among them — train only a small set of added weights, and community benchmarks put a fraction of a percent of trainable parameters within 90–95% of full fine-tuning's results. What used to be a datacenter project is now a strong-single-GPU project.
You need less data than you fear — sometimes. Meta's team reports ChatGPT's accuracy on Reddit-comment sentiment analysis jumping from 48% to 73% with one hundred examples, and a Phi-2 model on financial sentiment going 34% to 85%. Their rule of thumb: when baseline accuracy is under 50%, a few hundred examples often yields a large jump. Above that, returns flatten and effort shifts to data curation.
It can forget. The community has a name for it — catastrophic forgetting: a model tuned hard on one domain can shed general abilities it used to have. And there's a privacy angle that runs opposite to intuition: fine-tuning bakes your data into the weights, from which it can be partially extracted — for regulated industries, Databricks notes that keeping sensitive documents in a permissioned retrieval layer (RAG) is the more governable design, not less.
A three-question filter
Meta's full framework asks eight questions. In practice, three of them do most of the work:
| Ask | Answer | Lane |
|---|---|---|
| What's broken — facts or behavior? | Facts/ freshness/ citations | RAG (prompts first) |
| Tone/ format/ consistency | Fine-tuning (prompts + few-shot first) | |
| How much labeled data do you have? | Under ~a few hundred pairs | Prompts + RAG; fine-tuning likely starves |
| Hundreds+, stable task | Fine-tuning becomes viable | |
| How often does the knowledge change? | Daily/ weekly | RAG — retraining chases a moving target |
| Rarely/ never | Either; fine-tuning's stability pays off |
If you take one habit from this article, take the first row: before choosing a technique, name the complaint in one sentence — "it doesn't know X" or "it doesn't act right." The sentence picks the lane for you.
Combined beats either, when it's justified
Both Meta and Databricks converge on this: in production, hybrid systems often outperform either technique alone. The classic pipeline — a medical assistant, say — fine-tunes the model on medical literature so it speaks the language, then layers RAG on top so it retrieves the current clinical guidelines. Behavior from the weights, knowledge from the index, citations from the retrieval.
The catch: two systems means two sets of pipelines, evaluations, and failure modes. My ordering advice for teams with finite engineering time: prompts and few-shot until they creak, RAG for whatever knowledge gap remains, fine-tuning when a measured behavior gap survives all of that. "Hybrid is best" is true, and it's also how projects quietly double their maintenance bill — go hybrid because both halves earn their keep, not because a diagram said so.
Quick answers
Which is better, RAG or fine-tuning? Neither, universally: RAG for knowledge (fresh, cited, private), fine-tuning for behavior (tone, format, consistency). The popular "start with RAG, move to fine-tuning" ordering is a decent default but, as Meta argues, too simple — the strongest production systems often run both.
If I fine-tune, do I still need RAG? Usually yes, if your facts change. Fine-tuning doesn't reliably inject new knowledge, and its knowledge is frozen at training time; retrieval keeps answers current and citable.
What is LoRA, in one sentence? A parameter-efficient fine-tuning method that trains a small set of added weights instead of all parameters — roughly 0.1–1% of the trainable load for 90%+ of the benefit on typical tasks.
How much data do I need to fine-tune? Fewer than you'd guess when the baseline is weak — Meta reports 25-point gains from 100 examples; a few hundred pairs is a reasonable entry point. More important than volume: examples that reflect what production traffic actually looks like.
Is fine-tuning safer for private data than RAG? Counterintuitively, no. Fine-tuning bakes data into weights, which can leak back out; RAG keeps documents in a permissioned layer you control and audit. Regulated industries generally prefer the retrieval design.
The next time the room splits three ways, ask the question underneath the debate: what exactly is wrong with the outputs? Name the problem, and the ladder — prompts, examples, retrieval, training — tells you which rung to stand on. Start low, measure, climb only when the data says so.