Continued Pretraining — how knowledge actually enters weights

Twice now I've told you "fine-tuning can't reliably add facts" (Topics 12 and 14). Fair question: then how did the facts get in at all? Answer: pretraining — massive-scale next-token prediction on raw text. And that method is available to you too, at smaller scale. It's called continued pretraining (or domain-adaptive pretraining): take a trained model and resume plain next-token prediction on a large corpus of raw domain documents.

Note what's different from everything in Lesson 3: no instruction pairs, no chat template, no roles. Just documents — papers, contracts, codebases, wiki dumps — fed as raw text, exactly like stage 1 of the pipeline diagram. The model isn't learning to answer; it's absorbing the domain's vocabulary, style, and factual regularities into its knobs the same way it originally absorbed the internet.

The threshold that decides everything: scale. Knowledge enters weights through repeated statistical exposure, and that takes volume:

• Below ~50–100 million tokens of domain text → continued pretraining barely moves the needle. Use RAG. (For calibration: 100M tokens ≈ tens of thousands of long documents. Your company wiki is nowhere close.)
• Hundreds of millions to billions of tokens → now it genuinely works. Real examples: Code Llama = Llama 2 + continued pretraining on ~500B tokens of code; BloombergGPT-style finance models trained on enormous financial corpora; medical and legal domain models; and — directly relevant to you — language adaptation: taking an English-heavy model and continued-pretraining on, say, a large Urdu corpus is the standard recipe for building strong models in underserved languages. (With the Lesson 1 caveat: if the tokenizer shreds Urdu into tiny pieces, teams sometimes extend the vocabulary with new tokens and train their embeddings — powerful, but an advanced surgery with real failure modes.)

The price: it breaks the model's manners. You're training on raw documents, so the model drifts back toward being an autocomplete engine — instruction-following and chat behavior erode, because nothing in the corpus reinforces them. Continued pretraining effectively rewinds the model toward base-model behavior, but now domain-smart. Which forces the full recipe — I call it the sandwich:

instruct model (or base)
  → continued pretraining on domain corpus     [knowledge in, manners out]
  → SFT again on instruction data              [manners back]
  → optional preference tuning                 [polish back]

Budget for the whole sandwich, not just the middle. Teams that skip the re-SFT step ship a domain-brilliant model that responds to "summarize this contract" by continuing it.

Forgetting, again, and its standard fix: replay. Trained purely on legal text, the model doesn't just lose manners — it degrades at math, code, and general knowledge. The mitigation is mixing general data back into the domain corpus — commonly 10–50% general text alongside the domain text — plus a gentle learning-rate schedule (a brief warmup ramping the rate up, then a slow decay; sudden full-strength updates on a new distribution shock the knobs).

Where each method sits — the decision table you'll use for years:

	Prompting	RAG	Fine-tuning (SFT)	Continued pretraining
Changes	Nothing (context only)	Nothing (context only)	Behavior/form	Knowledge/distribution
Data needed	0	Your documents, any amount	500–50K examples	100M–billions of tokens
Cost	~0	Low	Modest	Serious (GPU-weeks+)
Facts update when world changes?	Instantly	Instantly (re-index)	Retrain	Retrain
Best for	Instructions	Facts, freshness	Format, style, reliability, cost	New domain, new language

Notice the "facts update" row — it's the quiet killer argument for RAG in most products: the world changes, your index updates tonight, while baked-in knowledge goes stale until the next expensive run.

---