Fine-Tuning vs RAG: A Decision Framework

By AI Agent Hub Editorial Desk · Review method · Corrections

Teams often ask whether a model should be fine-tuned “on the codebase” or connected to retrieval. That framing hides the real decision. First identify the failure: missing current facts, inconsistent behavior, poor task formatting, weak search, or a model that cannot perform the task even with good context.

What each approach changes

Choose retrieval when knowledge changes

Retrieval is usually the first experiment for active repositories and changing documentation because the source remains outside the model. A good system does more than place files in a vector database: it respects access control, chunks by meaningful structure, combines lexical and semantic search when useful, reranks candidates, and shows the evidence used. For code, symbol-aware search and dependency context often matter more than the choice of embedding model.

Choose fine-tuning when behavior repeats

Fine-tuning is worth evaluating when the desired input-output pattern is stable and prompt examples are insufficient or too expensive. Examples include a consistent classification taxonomy, a constrained transformation, a house style, or a specialized tool-selection pattern. Training data must represent the real distribution, include difficult negatives, and exclude secrets or material you lack permission to use.

Do not skip the baseline

Before building either system, create a fixed evaluation set and measure a strong prompt-only baseline. Record task success, factual support, latency, token use, and failure severity. Then test retrieval and fine-tuning separately. Without this sequence, a hybrid can become expensive while hiding which component helped.

Retrieval failure modes

• Wrong evidence: improve indexing, filters, queries, and reranking before changing the generator.
• Missing permissions: enforce authorization before retrieval, not after text reaches the model.
• Context overload: fewer high-quality passages can outperform a large dump.
• Stale index: monitor source-to-index delay and deletion propagation.
• Instruction injection: retrieved text is untrusted data and must not override system policy.

Fine-tuning failure modes

• Memorizing bad examples: deduplicate, review labels, and reserve a clean evaluation set.
• Behavior drift: retest after base-model or dataset changes.
• False factual confidence: training does not make changing facts reliably current.
• Hidden maintenance: dataset versioning, safety review, training runs, deployment, and rollback all need owners.

When a hybrid is justified

A hybrid is useful when evaluation demonstrates two independent needs: stable behavioral adaptation and current external evidence. For example, a tuned model may consistently produce an organization's incident schema while retrieval supplies the latest runbook and service metadata. Keep the components measurable: log which evidence was retrieved and compare the tuned model against the same retrieval context.

Practical decision sequence

1. Write 50–200 representative test cases with expected outcomes and risk labels.
2. Measure a prompt-only baseline with a current capable model.
3. If failures come from missing facts, test retrieval.
4. If failures come from repeated behavior, test fine-tuning.
5. Add the second method only if it produces a measurable improvement.
6. Re-evaluate cost, latency, privacy, and operational ownership before production.

Design a fair evaluation before choosing

Use one frozen test set across the prompt-only, retrieval, fine-tuned, and hybrid candidates. The following 120-case plan is a template, not a statement of measured results:

Track supported-answer accuracy, retrieval recall at the chosen context limit, citation correctness, category accuracy, abstention precision, p50/p95 latency, and cost per accepted task. Report every metric by slice; an overall average can hide a severe permission or freshness failure.

Attribute errors to the right component

1. Corpus coverage: did the authoritative source exist and was it eligible for this user?
2. Retrieval: did a relevant passage rank inside the context budget?
3. Generation: did the answer follow the supplied evidence?
4. Behavior: did it use the required taxonomy, tone, and output contract?
5. Policy: did deterministic code prevent unauthorized disclosure or action?

Labeling failures at these boundaries prevents a team from fine-tuning around a broken index or rebuilding search to solve a formatting problem.

Compare total operating cost

Use a workload model rather than a one-time training invoice:

monthly retrieval cost = searches + reranking + added input tokens + index operations
monthly tuning cost    = training runs + data review + evaluations + tuned-model inference
total cost per accepted task = (monthly platform + operations + review cost) / accepted tasks

Also record engineering ownership. Retrieval requires ingestion monitoring, access-control synchronization, deletion handling, and citation UX. Fine-tuning requires dataset governance, training reproducibility, model-version compatibility, regression evaluation, and rollback. Provider availability and supported base models change, so verify the current vendor documentation before committing to a training pipeline.

Data governance checklist

• Document the legal and contractual basis for indexing or training on each source.
• Separate training, validation, and final test examples by entity or time where leakage is possible.
• Deduplicate near-identical examples before splitting the dataset.
• Redact secrets and unnecessary personal information from prompts, traces, and training files.
• Make source deletion propagate to retrieval; define what deletion means for trained artifacts and backups.
• Version the corpus snapshot, chunker, embedding or search settings, prompt, dataset, grader, and model.

Phased rollout with stop conditions

1. Ship a prompt-only baseline behind logging that excludes unnecessary sensitive content.
2. Evaluate retrieval offline, then shadow it without showing answers to users.
3. Release retrieval to a small group with citations and a visible fallback.
4. Train only after the behavioral error slice is large and stable enough to justify it.
5. Compare the tuned candidate against the same retrieval context and held-out set.
6. Stop the rollout if restricted-document exposure, unsupported-answer rate, or severe regression exceeds the predefined threshold.

Primary references

• OpenAI Retrieval guide
• OpenAI Fine-tuning guide
• Anthropic contextual retrieval engineering note

The running cost on the RAG side of the decision

The fine-tuning versus RAG decision is usually framed as a capability trade-off, but the two have completely different cost shapes: fine-tuning is an upfront cost plus a per-token delta, while RAG is a pure per-query cost that scales linearly with traffic. Below, the RAG side: one query at 25K input, 15K cached, 2K output. Of the 25,000 input tokens, 15,000 are billed at the cache-read rate and 10,000 at full input rate.

At thirty thousand queries a month the small models keep retrieval in the low tens of dollars, which is the number a fine-tuning run has to beat. Fine-tuning wins on cost only at volumes or latencies where the per-query path itself is the constraint.

Rates verified against provider documentation on September 18, 2026. Promotional rates expire, so re-check before budgeting: LLM API cost planning · September 2026 pricing update. Run your own numbers in the cost calculator.