RAG vs Fine-Tuning vs Continued Pretraining

The question shows up in almost every enterprise AI scoping call we run: “Should we do RAG, fine-tune the model, or continued-pretrain it on our data?”

It’s the wrong question, and the wrong question has a cost. Teams that pick “one of the three” end up rebuilding twice — first when RAG can’t produce a fixed JSON response the downstream system needs, second when fine-tuning can’t answer a question about a policy that shipped last Tuesday. The 2026 enterprise pattern is not to pick. It’s to know which problem each one solves and stack them.

This post is the cut we use with clients when scoping generative AI engagements. It covers what each approach actually changes about the model, the real production cost of each, the failure modes vendor blogs skip, and the composed architecture that’s winning enterprise deployments right now.

What each approach actually changes

Most comparison articles treat RAG, fine-tuning, and continued pretraining as three flavors of the same problem. They aren’t. They intervene at different layers.

RAG changes what the model knows at inference time. The model weights don’t move. You build an index of your documents, retrieve the most relevant chunks per query, and stuff them into the prompt. Update the index and the model “knows” the new thing tomorrow. RAG is a knowledge layer bolted onto a fixed model.

Fine-tuning changes how the model responds. You update a small fraction of the weights on your labeled examples. The model doesn’t magically know new facts — it learns your response shape, your terminology, your refusal patterns, your JSON schema. Fine-tuning is a behavior layer.

Continued pretraining changes what the model has read. You keep training the base model on a large corpus of raw unlabeled domain text — legal briefs, clinical notes, chip design specs. The model’s underlying representation of language shifts toward your domain vocabulary. Continued pretraining is a vocabulary and world-model layer.

Say it this way and the substitution framing falls apart. Knowledge, behavior, and world-model are different things. Trying to solve a knowledge problem by shifting behavior — fine-tuning the model on your FAQ — is one of the most expensive mistakes we see. It works in the demo. It rots in production the first time the FAQ changes.

The rule of thumb, honestly

We’ve run this decision on somewhere north of 40 enterprise engagements in the last 18 months. The empirical split is not 33/33/33:

• ~85% of enterprise use cases: RAG is right. Customer support, knowledge base search, contract analysis, policy Q&A, sales enablement, internal chat over documents — anything where the answer depends on facts that change. Almost every “we want an AI assistant for X” starts here.
• ~12% of use cases: fine-tuning earns its keep. Structured extraction with strict schemas, classification into custom taxonomies, tone/format-critical generation (marketing copy in a brand voice), and code generation in an internal DSL. The through-line is that the shape of the response matters more than the facts inside it.
• ~3% of use cases: continued pretraining is worth it. Highly specialized scientific corpora — pharma R&D, semiconductor design, structural engineering — where the domain vocabulary is far enough from web text that base models tokenize it badly and reason about it poorly. Roughly one in every thirty engagements.

If your split doesn’t look like this, either your problems are unusually exotic or someone is selling you the wrong tool. Continued pretraining in particular is a technique enterprise teams keep asking about because vendor blogs made it sound like the sophisticated choice. It’s the least often correct one. In AI consulting engagements we push back on it hard unless the domain justifies it.

Real production cost, not vendor demo cost

The cost claims in most comparison posts stop at “training run cost” and skip everything after. Enterprise production cost is the sum of build, run, and evolve. Here’s the shape we see on real workloads.

RAG all-in for a production knowledge assistant — 500K documents, 10,000 queries/day, one refresh a week — lands at $4,000–$9,000/month. That’s embedding costs (usually the smallest line), vector store ($800–$2,500 depending on Pinecone vs a self-hosted Qdrant/Weaviate), LLM inference (the largest line, $2,000–$5,000), retrieval-eval pipelines, and the ops burden of keeping the index in sync with source systems. The last two items are what people forget. An index that drifts against source is worse than no index at all.

Fine-tuning with LoRA/QLoRA adapters has become genuinely cheap. A 7B-class model fine-tuned with QLoRA runs on a single H100 for well under $100 per training run. The adapter file is 50–200 MB, merges into the base weights at load time, adds zero inference latency, and can be swapped per tenant if you’re building a multi-tenant product. Full fine-tuning of the same model would cost $2,000–$10,000 per run and produce a multi-gigabyte artifact. There is almost no enterprise case in 2026 where full fine-tune beats LoRA.

The trap in fine-tuning cost isn’t the training run — it’s the eval suite. A serious fine-tune needs a golden set of 500–2,000 examples, held-out regression tests, and a rerun protocol every time the base model version bumps. Building the eval suite typically costs 3–5x the training runs it sanctions. Teams that skip it discover why six months later when their fine-tune quietly regresses on the workload that mattered.

Continued pretraining costs real money. For a 7B base model on 10–50 billion tokens of domain text — the low end of what actually shifts behavior — you’re looking at $30,000–$150,000 in compute for one pass, plus a data preparation pipeline that typically costs more than the compute. Then you need to re-do your fine-tune on top of the new checkpoint. Then evaluate that everything downstream still works. Then re-do it when the frontier base model advances two generations, because the checkpoint you produced is now behind. This is why continued pretraining is only defensible when domain vocabulary is truly distant from the base — the frontier moves faster than your checkpoint can amortize.

The failure mode each has

Vendor material tends to describe each approach’s strengths. The failure modes are more useful.

RAG’s failure mode is index rot. The index was built once, is now three months stale, retrieves the old policy document ahead of the new one, and the model confidently cites the wrong version. The fix isn’t the model — it’s a data pipeline that keeps the index in lockstep with the source of truth, with freshness monitoring on both. This is the piece that turns a RAG proof-of-concept into a production system, and it’s why we treat it as a data engineering problem as much as an AI problem.

Fine-tuning’s failure mode is version drift. You fine-tuned on gpt-4o. Six months later everyone’s on the next generation and your adapter no longer applies. Or the base model provider deprecates the checkpoint underneath you. Fine-tunes have a lifecycle problem RAG doesn’t: they’re coupled to a specific base version and need re-doing when that version moves. Serious deployments budget for this. Casual deployments discover it as an outage.

Continued pretraining’s failure mode is catastrophic forgetting. You trained on 20 billion tokens of legal briefs and the model got noticeably better at legal reasoning — and noticeably worse at basic math, code, and general instruction-following. The regression is real, it’s documented in the literature, and mitigating it requires interleaved general-domain data during the CPT run, which raises cost and complexity again. This is a large part of why enterprise CPT rarely pays off on generalist base models: you’re spending a lot to specialize and losing capability you were paying the base provider for.

The 2026 decision matrix

If the primary problem is…	Right approach	Wrong choice teams often make
Answering questions from a changing knowledge base	RAG	Fine-tuning on the docs — updates require retraining
Producing a specific JSON schema or fixed response shape	Fine-tuning (LoRA)	RAG + prompt engineering — brittle and expensive per token
Classifying into an internal taxonomy of 20+ labels	Fine-tuning (LoRA)	Few-shot prompting — accuracy plateaus, cost scales badly
Writing in a strong internal brand voice or tone	Fine-tuning (LoRA)	Long system prompts that eat context and drift
Reasoning over a scientific corpus with alien vocabulary	Continued pretraining (rare)	RAG alone — retrieval helps, but tokenization still fails
Enterprise Q&A on policies, contracts, tickets	RAG	Fine-tuning — locks answers to a moment in time
Multi-tenant customization per customer	LoRA adapters per tenant, shared base	Full fine-tune per tenant — cost and ops explode
Compliance-critical answers that need citations	RAG with source attribution	Fine-tune — no traceable source of truth

The matrix is deliberately narrower than “here are ten factors to score across three columns.” Most enterprise problems collapse cleanly to one of these rows once you name the primary problem honestly.

The composed pattern that’s actually winning

The right question in 2026 is not “which one” but “which combination.” The pattern we deploy on serious enterprise engagements now looks like this:

1. Base model: a strong frontier model — Claude, GPT, Llama depending on the compliance and cost posture. Not fine-tuned, not continued-pretrained. You want the frontier’s general capability.
2. RAG layer for knowledge: production-grade retrieval over the domain corpus, with the index refresh pipeline, freshness monitoring, and eval loop treated as first-class data infrastructure.
3. LoRA adapter for behavior: a small fine-tune on 500–5,000 labeled examples of the exact response shape you need — the JSON schema, the citation format, the refusal pattern, the tone. Swappable per use case.
4. Continued pretraining only if the domain demands it: rare, and if you’re doing it, you’re doing it once and building everything above it on the resulting checkpoint.

RAG owns knowledge. LoRA owns behavior. Continued pretraining, if present at all, owns vocabulary. Each layer’s failure modes are independent, so each can be operated by the right team — data engineering for the RAG index, ML for the fine-tune, platform for the base model choice. This is the enterprise AI stack we ship in production, and it’s what our Arcastra agent platform is built on top of. The composability is the point: swapping any single layer doesn’t require rebuilding the others.

The wrong version of the pattern is picking one and forcing it to do everything. RAG-only deployments hit a wall on structured output. Fine-tune-only deployments hit a wall on knowledge freshness. Continued-pretrain-only deployments hit a wall on frontier obsolescence. All three walls are avoidable if the architecture treats each layer as what it is.

Where the ground is shifting

Three things worth watching over the next 12–18 months, in roughly this order:

Long context reduces some RAG surface area. Frontier models with million-token context windows can hold entire policy manuals or contract sets in-prompt. For small enough corpora — a few thousand documents — retrieval starts to look overengineered. RAG’s edge remains on the large, dynamic corpora where indexing beats context. But the “RAG for anything document-shaped” default is loosening.

LoRA adapter marketplaces. Serving infrastructure like vLLM, Text Generation Inference, and cloud model hubs are treating adapters as first-class artifacts you can hotswap per request. Multi-tenant products that ship a distinct adapter per customer become genuinely tractable. This is where fine-tuning quietly becomes a scale advantage rather than a per-project cost.

Domain-adapted base models from providers. Anthropic, OpenAI, and open-source communities are shipping models pretrained on medical, legal, and code corpora. When your domain has a good provider-shipped base, DIY continued pretraining stops being defensible. The remaining CPT budget is for domains too narrow for provider interest — pharma R&D, defense, industrial process control.

What this means for a 2026 GenAI build

If you’re scoping an enterprise GenAI project, the sequence we’d suggest is boring on purpose:

1. Start with RAG on a frontier model. No fine-tune, no CPT, real retrieval infrastructure. Get to a working proof.
2. Instrument the eval loop — a golden set of 200+ queries with expected answers, plus retrieval-quality metrics, before you touch the model.
3. Add a LoRA adapter only when the eval shows a specific behavior gap the base model can’t meet with prompting alone. Typical trigger: structured output failures, or a tone/format that costs too many tokens to prompt.
4. Consider continued pretraining only if a specific vocabulary or reasoning gap survives step 3 — and only after you’ve checked whether a domain-adapted provider model already exists.

Most projects end after step 3. That’s fine — it’s the correct answer for most enterprise AI. When teams tell us they need “all three,” we usually find they need one of them well-run, and the demand for the other two dissolves once the first one works.

If you’re mid-scoping and trying to figure out which layer your problem actually needs, our AI strategy engagement typically maps this in the first two weeks, with a written recommendation that names the approach, the base model, and the eval plan. It’s the least glamorous week of the project. It’s also the one that decides whether the model ships or becomes another 80% demo.