Module 3 · Fine-Tuning Methods

Lesson 6Preference Algorithms, Portable Models, and Your First Real Fine-Tune

3 topics14 min readTopics 27–29

Welcome to Lesson 6 — the back half of Module 3. Today the preference pairs you learned to build in Topic 16 finally get consumed by actual algorithms, and at the end, everything from four lessons converges into one runnable pipeline. Let's finish this module properly.

In this lesson

  1. 27RLHFthe heavyweight that started it allRLHF (Reinforcement Learning from Human Feedback) is the stage-3 method that turned GPT-3 into ChatGPT — historically, the technique that…4 min
  2. 28DPOthe shortcut that took overIn 2023, a paper arrived with a title that summarizes itself: "Direct Preference Optimization: Your Language Model is Secretly a Reward…4 min
  3. 29GGUFthe format your model wears to leave homeEverything so far lives in the training world: Hugging Face repos, a folder of files — model.safetensors shards, config.json, tokenizer…6 min

Start with Topic 27: RLHF, or browse all 90 topics.