Stanford OnlineStanford CME 295 Transformers & LLMs | Fall 2026 | Lecture 3 - LLM training
Stanford Online
Training goals 0:05
The lecture shifts from model design to training. It recaps scaling laws, the need to avoid undertraining, mixture of experts, and newer transformer tweaks such as RoPE, sliding-window attention, grouped query attention, and normalization. The main focus now is how to train a language model so it matches the end use.
Pretraining and data 3:32
Training usually starts with pre-training, where a decoder-only model learns language from large mixes of text, code, math, and science. The task is next-token prediction. The model sees earlier tokens and learns to predict the next one, with masked self-attention keeping each prediction limited to what comes before. This stage uses cross-entropy loss, and the same idea can be written as maximizing the probability of the training sequence, then taking the log and averaging across tokens. The section also notes bits per byte as a tokenizer-free way to compare losses across different tokenizers. Data choice matters too. Sources can come from the web and code repositories, and large corpora may be repeated when data is limited. Work like DataComp LM and data-mixture studies tries to separate the value of data quality from model design and to find useful blends of web and code at scale.
Longer context training 24:00
Training shifts from short sequences to much longer ones, sometimes 128k or 200k tokens. The model keeps RoPE position encoding, but the frequencies are tweaked so long distances stay within ranges it has already seen. This helps when you want the same model to handle far longer inputs at inference time.
Instruction tuning 32:01
After pre-training, the model still tends to autocomplete instead of answering. Supervised fine-tuning changes that by training on instruction and response pairs, while still using next token prediction. You give the prompt as input and train the model to predict only the answer, so it learns an assistant-like style.
LoRA efficiency 40:02
A model does not always need all of its weights updated to adapt to a new task. LoRA, short for low rank adaptation, keeps the pre-trained weights and learns small low-rank matrices instead. That cuts training and storage costs, and it can be used for separate tasks like spam, sentiment, and translation with the same base model.
Preference data 47:00
The focus shifts to human preference data. A general model can answer questions, but you may still want to tell it what not to say. To do that, you collect pairs of answers for the same prompt, with one response preferred and one rejected. Pairwise judgments are easier for people than absolute scores, and they are usually the common choice.
Reward modeling 52:01
The first step is to train a reward model that scores a prompt and response as good or bad. The lecture uses the Bradley Terry form, where the chance that one answer is preferred over another is the sigmoid of the difference between their rewards. The model is trained on winning and losing responses with a pairwise loss, often built from a log of that preference probability.
Aligning with rewards 1:01:31
The second step is reinforcement learning from human feedback, or RLHF. The reward model is frozen, and the language model is updated so its answers earn higher reward while staying close to the reference model. That restraint matters because the reward model is not perfect, and pushing too far can lead to reward hacking, where the model learns to please the reward model without truly improving.
Preference tuning 1:05:01
The reward model stays fixed while you update only the LLM. That is the point to remember from reward modeling and RLHF. DPO, or direct preference optimization, tries to do the same job in a supervised way. It pushes the model to raise the chance of the winning answer and lower the chance of the losing one, usually starting from a supervised fine-tuned reference model.
Reasoning models 1:12:01
Reasoning is treated as the ability to solve a problem. Simple knowledge questions are different from tasks like working out a bear’s age from its birth year. Chain-of-thought prompting can elicit step-by-step work, but the goal is for the model to do that on its own. These models are now common, and the reasoning chain is counted as part of the output, so it can affect cost and is often hidden for safety and training reasons.
Verifiable training 1:19:02
For math and coding, rewards can be checked against ground truth or test cases. That lets you train with a reward for a correct answer and for showing reasoning. GRPO, or group relative policy optimization, samples several replies, compares them within the group, and pushes the model toward the better ones while staying close to the base model. This improves performance, and the reasoning chains tend to get longer as training goes on.
Dense on-policy signals 1:23:32
A key limit here is that sequence-level rewards are coarse, since the whole output is only right or wrong. Earlier token-level methods gave denser feedback, but they used off-policy data, meaning outputs the model did not produce itself. On-policy distillation tries to combine both strengths. The student generates the response, a teacher scores each token, and deviations are penalized. A variant is on-policy self-distillation, where the same model serves as teacher and student.
Training stages in practice 1:27:02
The model training pipeline is broken into pre-training, mid training, supervised fine-tuning, and reinforcement learning. Pre-training starts with filtered web text, then adds coding and reasoning data. Mid training shifts toward longer, higher quality material, including code reviews, issues, and even books, so the model can handle longer context. Supervised fine-tuning then narrows the loss to the answer side of a prompt and leans hard on helpfulness, reasoning, and tool-use traces hidden inside think tokens. Reinforcement learning is split into reasoning RL, agentic RL, and general RL, with verifiable math, test-based bug fixing, and expert judgments all used to shape the answer.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
