Stanford OnlineStanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs
Stanford Online
RL training overview 0:06
Today shifts to reinforcement learning methods for post-training LLMs. The lecture starts with a warning that some math is coming, then returns to the training stages you already know: pre-training on trillions of tokens with cross-entropy loss, supervised fine-tuning to make the model useful, and preference tuning and reasoning, where RL-based methods enter.
Rewards and gradients 7:03
The setup is an agent, a state, an action, and a reward, with the policy pi theta choosing what comes next. In LLMs, the prompt can be the state and the response the action. Rewards can come from human preferences through a reward model and Bradley-Terry style training, or from verifiable checks such as math answers or code tests. The core objective is to maximize expected reward, and the gradient is rewritten with a log term so actions with positive reward become more likely and negative ones less likely.
Policy updates 23:30
A response is sampled from the current LLM, scored with a reward, and the model is updated by gradient ascent. This setting is on policy, so you must generate fresh completions for every update. That makes training expensive, because token generation happens one step at a time. To cut cost, the same completion can be reused by reweighting it with an importance sampling ratio from a fixed policy.
Challenges and PPO 33:00
Three problems show up in the plain setup. Rewards without a baseline can make gradients noisy. Completion-level rewards do not give token-level credit. The policy can also drift too far from the model you started with, which risks reward hacking and losing useful skills. Proximal policy optimization, or PPO, addresses this by using advantages and keeping the new policy close to the reference.
Advantage and policy control 48:00
The value model is trained with the policy, while the reward model is trained first. The point is to estimate expected reward under the policy you are updating. Because completions are costly, you reuse one batch and compare the current policy to the old one with an importance sampling ratio. That ratio is clipped so a positive advantage does not keep pushing the policy to raise the same action forever, and a negative one does not push it down without limit.
KL penalty and GRPO 58:31
A second guardrail is KL divergence between the current policy and a fixed reference policy. Beta sets how hard you resist moving away from that reference. The clip and the KL term work together, though papers mix them in different ways. The main drawback is the value model. GRPO avoids it by sampling several responses for one prompt, scoring each one, and turning those scores into relative advantages by subtracting the group mean and dividing by the group standard deviation.
GRPO and PPO compared 1:11:01
GRPO uses a reward model, a reference model, and no value function. The policy is trained with clipping and a KL penalty against the reference. PPO instead adds a value model that estimates token-level expected reward and is trained with the policy. That makes PPO heavier to run, because the two models must stay in sync while the policy is updated.
Length bias and fixes 1:18:30
GRPO compares each completion to the group mean reward, but its normalization can favor longer bad answers when performance plateaus. Papers such as DAPO and Dr. GRPO change or drop that normalization. Another issue is that outcome-only rewards ignore bad intermediate steps, so process supervision would need token labels, which is expensive. A proposed next step is distillation with reverse KL.
Teacher comparison 1:32:33
Reverse scale divergence is forgiving when teacher and student differ on low-probability tokens, but it strongly punishes cases where the student assigns weight to a token the teacher sees as unlikely. That makes the student mode seeking, so it learns the teacher’s peaks. In on-policy distillation, the student’s own continuation is fed back to the teacher at each token. The signal is dense, works under compute limits, and can be reused through self distillation or split across multiple domain teachers.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
