Every LLM project reaches the same fork: the base model isn't doing what you need. Three roads lead away from that fork — prompt engineering, retrieval-augmented generation, and fine-tuning — and teams routinely pick the wrong one because they reason from familiarity rather than from the structure of the problem. After shipping all three into production repeatedly, here is the framework we actually use.
Start with what kind of gap you have
The mistake is treating these as interchangeable levers of increasing power. They aren't. They solve different categories of failure:
- Prompt engineering fixes instruction gaps. The model knows how to do the task; you haven't specified it precisely. Format, tone, steps, edge-case handling — these are specification problems.
- RAG fixes knowledge gaps. The model can't know your contract terms, last week's policy update, or your product catalog. No amount of prompting injects facts the model never saw.
- Fine-tuning fixes behavior gaps. The model needs to internalize a style, a structured output dialect, or a domain reasoning pattern that's too complex or too token-expensive to specify in every prompt.
Diagnose the gap before choosing the tool. A surprising number of "we need to fine-tune" conversations end after a corpus audit reveals a knowledge gap that RAG closes in a tenth of the time and cost.
The order of operations
In production, the sequence almost always runs: prompt first, RAG second, fine-tune last — not because prompting is weak, but because each step is cheaper to test and easier to reverse than the next. Prompt changes deploy in minutes and roll back instantly. RAG changes require pipeline work but leave the model untouched. Fine-tunes create a model artifact you must version, evaluate, host, and eventually re-train when the base model improves — which it will, on someone else's schedule.
A fine-tune is a liability you maintain. A prompt is a config change. Choose your liabilities deliberately.
Where each one quietly fails
Prompt engineering fails at scale of variation. Twenty edge cases become a 4,000-token instruction block that degrades performance and inflates cost. When your prompt starts resembling a legal document, the model is telling you the behavior should live elsewhere.
RAG fails at retrieval, not generation. In our production audits, 70–80% of bad RAG answers trace to the retriever surfacing the wrong context — not the model misreading good context. Teams burn weeks on prompt tweaks when the fix is chunking strategy, hybrid retrieval, or corpus hygiene. Measure retrieval recall before touching the prompt.
Fine-tuning fails silently through drift and lock-in. A fine-tune captures a snapshot of desired behavior. Your domain moves, your base model deprecates, and the artifact decays. Without a regression eval suite, you won't notice until users do. We've also watched teams fine-tune knowledge into models — it sort of works in the demo and fails unpredictably in production, because gradient descent is a terrible database.
The hybrid reality
Mature systems usually combine at least two: RAG for current knowledge with a carefully engineered prompt, sometimes plus a small fine-tune (or LoRA) for output structure when JSON-mode prompting proves brittle. The architecture question isn't which one — it's which gap does each component own, and how do you eval each independently so failures localize.
The decision in four questions
- Does the failure involve facts the model couldn't know? → RAG.
- Can a clearer instruction with examples fix it in under ~1,500 tokens? → Prompt.
- Is it a stable style/format/reasoning pattern, with 500+ quality examples available, that prompting can't hold? → Fine-tune.
- Is knowledge changing weekly? → RAG, and never fine-tune that knowledge in.
Whatever you choose, build the eval set first. All three approaches are easy to demo and impossible to manage without measurement — and the eval suite is the only artifact that survives all your future migrations.