Stanford OnlineStanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 5 - LLM Systems
Stanford Online
Why systems matter 0:05
LLMs build on the transformer decoder and turn text into text. They are scaled up in model size and data, often to billions or even trillions of parameters and trillions of tokens in pre-training. That makes hardware and infrastructure central, because you need to train and serve these models in practice, not just describe them on paper.
GPUs and attention 2:01
The decoder is mostly masked self-attention and a feedforward network, which means a great deal of matrix multiplication. GPUs are built for that kind of work because many products can be done in parallel across many cores. The lecture then focuses on what limits GPU use: memory is tight, compute is costly, and the attention matrix can become huge, so the goal is to fit the model and make it fast.
Iterative softmax 24:03
The softmax for attention is computed one key at a time. For each score Si and value Vi, you update a running numerator and denominator instead of summing the whole row first. This lets you recover the same weighted result by dividing the final accumulated numerator by the final accumulated denominator. The point is the iterative form, not a new approximation.
Rescaling for stability 29:01
Exponentials can get too large, so the running totals are rescaled by a factor based on the largest score seen so far. The old sum is adjusted with exp(M1 minus M2) when a new maximum arrives, which keeps the numbers safe. The same fused update cuts reads and writes to HBM, uses SRAM for small pieces, and makes forward and backward passes faster while still computing the exact softmax.
FlashAttention details 47:00
The recomputation is for training, because you need the gradient there. At inference time you do not need to run the same work again. The main change from the vanilla case is that you do not have to keep the whole attention matrix in memory. You can update the numerator and denominator step by step instead of waiting for the full row, which avoids storing an n by n matrix. That also cuts reads and writes to HPM, which is slower than SRAM.
Why more GPUs 51:00
Training LLMs pushes memory past one GPU. You must store the model itself, the activations from the forward pass, the gradients from the backward pass, and extra optimizer state, such as what Adam keeps. The model size, the sequence length, and the batch size all make the activation memory grow. Since the decoder stacks are large and the data is large too, one device is often not enough.
Parallelism and cost 1:08:01
The work returns to the ways you split LLM work across GPUs. Data parallelism, tensor parallelism, pipeline parallelism, context parallelism, and expert parallelism can be mixed, but pushing parallelism too far wastes device time and raises communication cost. GPUs are rented by the hour, and moving parameters for some data-parallel setups is not free. The summary is simple: distribute parameters, activations, gradients, and states across GPUs, then use data and model parallelism to make large workloads fit and run faster.
Inference latency and memory 1:10:00
At inference time, the slow part is usually autoregressive decoding, not the initial prompt pass. A long input with a short output is often faster than a short input with a long output because the output is generated one token at a time. The key timing measures are time to first token, time per output token, and time to last token. Providers also expose input and output prices, and output is often about five times more expensive. The system goal is to stay fast and cheap within fixed limits.
Grouped attention and MLA 1:28:30
Grouped query attention cuts the number of key and value projection matrices from H to G. When G equals H, it is ordinary multi-head attention. When G equals 1, it becomes multi-query attention, which is too weak for good features. With H at 24 and G at 2, the memory drop is about 91 percent. The choice is a trade-off, so people tune G with grid searches and ablation studies. Multi-latent attention goes further by compressing what would be many key and value vectors into one small shared representation, then decoding it back when needed.
Speculative decoding 1:33:00
Speculative decoding speeds up generation by splitting work between a draft model and a target model. The draft model proposes several next tokens at once. The target model then checks them in parallel, which is cheap compared with generating them one by one. If the target is more confident than the draft, a token is accepted. If not, it is accepted with a ratio based on the two probabilities, or rejected and resampled from the leftover mass. The method keeps the target distribution exactly, and it helps avoid wasted work. Knowledge distillation and multi-token prediction are two ways to make the draft and target stay aligned.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
