TRL — the trainer zoo

Topic 45 of 90Module 5: Local AI Ecosystem2 min read

TRL ("Transformer Reinforcement Learning") is HF's official home of stages 2 and 3 — the library whose SFTTrainer ran your capstone. Its shape is simple: one trainer class per training method you've learned, all sharing the transformers/PEFT plumbing:

  • SFTTrainer — Topic 13–15 as code: chat templates and loss masking handled.
  • DPOTrainer — Topic 28, the triplet-eating shortcut.
  • RewardTrainer / PPOTrainer — Topic 27's classic pipeline, both halves.
  • KTOTrainer — the thumbs-up/down variant your future SaaS feedback button feeds.
  • GRPOTrainer — the modern star, worth thirty seconds of real explanation because it closes Topic 27's RLVR flag: GRPO (Group Relative Policy Optimization, the algorithm behind DeepSeek-R1) samples a group of answers per prompt, scores each with a verifiable reward (does the math check out? do the tests pass?), and treats each answer's advantage as its score relative to the group's average — eliminating PPO's fourth model (the value network) entirely. Cheaper, simpler, unhackable rewards: the recipe that trains reasoning models, runnable from a library you already have installed. Module 8 builds on exactly this.

Summary

One trainer per method from Module 3 — SFT, DPO, RLHF's parts, KTO, and GRPO for verifiable-reward RL — on shared HF plumbing.

Mental model

The gym where each of Module 3's training styles has its own station, same membership card.

Mistakes to avoid

Hand-rolling formatting/masking logic the trainers already do correctly (the bugs of Topic 15 are solved problems — let them stay solved), and grabbing PPOTrainer for preference tuning when DPOTrainer is the right first reach (Topic 28's whole argument).

Exercise

Read the TRL docs page for GRPOTrainer and write, in your own words, the reward function you'd define for a verifiable task you care about (unit tests for code? regex-checkable JSON validity?). One paragraph — and you've drafted your first RLVR experiment.