Every CTO looking to ship production AI eventually asks the same three-way question: agents, RAG, or fine-tuning? They solve different problems, cost different money, and fail in different ways. This is the framework we use before scoping any production AI engagement.
The one-sentence definitions
- RAG — retrieve relevant context, then let the model generate an answer grounded in it.
- Fine-tuning — train a model once on curated examples so it internalizes a task, tone, or format.
- AI agents — give the model tools (APIs, code execution, retrievers) and let it plan multi-step actions to achieve a goal.
These are complementary, not competing. Every serious production AI system we ship uses at least two of them together.
Sweet spots
RAG shines when…
- Answers depend on your private, changing knowledge.
- You need to cite sources or defend answers to auditors.
- Time-to-first-answer matters and you can't wait for a training run.
- The task is single-turn Q&A or lookup.
Fine-tuning shines when…
- You need consistent tone, style, or output format.
- You have a narrow classification / extraction task and want a small, fast, private model.
- You can produce (or already have) 500-50,000 curated training examples.
- The task is stable — retraining is expensive, so the target must be, too.
Agents shine when…
- The task requires multiple steps: search, then check, then act.
- The model must call tools (databases, APIs, calculators, code).
- The workflow benefits from LLM-style reasoning across those steps.
- You can accept variable latency and cost per task.
Cost profiles
| Approach | Upfront build | Per-query cost | Ongoing maintenance |
|---|---|---|---|
| RAG | Medium ($15-45k) | Low ($0.001-0.05) | Keep index fresh |
| Fine-tune | High ($25-90k + data) | Very low (small model) | Retrain when task shifts |
| Agents | Medium-high ($20-80k) | Medium-high ($0.05-1.00) | Tool schemas + eval |
Failure modes to know before you commit
RAG failures
- Bad retrieval → confident-sounding wrong answer.
- Bad chunking → the right passage is in the store but never surfaces.
- No reranking → top-K contains noise.
- No evaluation → nobody notices when accuracy drops.
Fine-tune failures
- Model becomes stale the moment your world changes.
- Training data leaked PII → GDPR / privacy problem baked into weights.
- Overfits the training set → looks great on eval, worse than base in prod.
- Can't audit why the model said what it said.
Agent failures
- Agent loops — burns budget circling on the same subtask.
- Tool-call errors compound — one bad step ruins the trajectory.
- Cost creep — a single agent run can cost 100x a single RAG query.
- Reliability drops sharply as step count grows past ~5.
The framework we use
We ask five questions in order:
- What does the user actually get? A cited answer? An action taken? A formatted document? Each of those points to a different architecture.
- How stable is the domain? If it changes weekly, avoid fine-tuning. If it's ancient and slow-moving, fine-tuning is fair game.
- How many discrete steps must happen to succeed? One = RAG. Two-three = RAG + tool calls. Four+ = an agent.
- What is the budget per task, in cents? Sub-1¢ = RAG or fine-tune. 5-20¢ = tool-using RAG. 20¢+ = agent territory (or rethink).
- How auditable does it need to be? High = RAG. Medium = agents with logged tool calls. Low = fine-tune.
Those five answers usually collapse to one architecture in about ten minutes.
Three architectures we actually ship
1. "Support agent" (RAG + light agent)
RAG for grounded Q&A, plus a small agent that can look up the customer's order or open a ticket if the answer is escalation. 90% of use cases fit here.
2. "Internal knowledge worker" (pure RAG)
Cite-source Q&A over wikis, docs, policies. No tools. Cheapest, safest, easiest to audit.
3. "Autonomous back-office task" (agent + fine-tuned tools)
Multi-step task — expense reconciliation, customer onboarding, contract triage. Agent orchestrates. Small fine-tuned models do the narrow classification / extraction steps cheaply.
What we rarely recommend
- A pure fine-tune to "add knowledge" — this is the #1 misconception. Use RAG.
- Agents with more than 5-7 steps and no human-in-the-loop.
- Agents on top of an expensive frontier model when 80% of the reasoning is cheap.
Recommendation
Start with RAG. Add a tool-using layer when a workflow needs it. Fine-tune only when RAG has genuinely hit its ceiling on style or on a very narrow task. That order minimizes cost, maximizes optionality, and gets working AI in front of users fastest.
If you're weighing this decision now, our AI engineering team ships all three architectures — usually in combination. Talk to us for a free 30-minute scoping call.
