Stanford Online

LLMs for Agent Builders: Context Engineering | Stanford CS329Z Lecture 2 with Michael Ryan: summary

YouTube summary13 sectionsWatch on YouTube ↗

This is an AI-generated summary of the YouTube video "LLMs for Agent Builders: Context Engineering | Stanford CS329Z Lecture 2 with Michael Ryan" (Stanford Online), made with Samuraize and published by Samuraize. It condenses the YouTube video into 13 titled sections you can read in a couple of minutes, each linking to the moment in the video it covers.

1
Filed under💻 Technology0 comments🍱 Add to trayReport
Study this

LLMs for Agent Builders: Context Engineering | Stanford CS329Z Lecture 2 with Michael Ryan

Stanford Online

切

Lecture framing 0:05

Michael opens the second CS329Z lecture and says he is still learning from the process, so questions are welcome. He places the class at the top of a language-model stack: model engineering at the bottom, adaptation in the middle, and applications such as agents, chatbots, and RAG at the top. The goal is to understand enough of the underlying design to choose models well and know their limits in long-context settings.

切

What language models do 4:01

A language model assigns a probability to a token sequence by multiplying the probability of each next token given the earlier ones, usually working with log probabilities for stability. That makes the model generative, since it can score and sample whole sequences. He shows decoding methods such as greedy choice, top-k, top-p, temperature, and beam search, and notes the trade-off between diversity, cost, and task fit.

切

Sampling and reasoning 15:00

Sampling matters wherever the model generates text. It is used for inference and for reinforcement learning, where you generate rollouts. Temperature also affects reasoning models, so it is another choice you may tune. The lecture then turns to language model history, from n-grams that counted token pairs to early neural models, recurrent networks, and LSTMs. Those older approaches are set aside here in favor of transformers.

切

Transformer mechanics 17:00

Transformers come from Attention Is All You Need and are usually decoder only in modern language models. The input tokens are embedded, given positional embeddings, and passed through attention. Queries, keys, and values are built with learned matrices, masked so the model cannot look ahead, then combined and projected back with a residual connection. Multi-head attention repeats this several times, then a feed-forward layer and stacked blocks produce the next-token prediction.

切

Why transformers won 28:30

The key advantage is parallelism. Attention lets a whole sequence be processed in one forward pass, while recurrence has to step token by token. That makes transformers far better for GPU training. The cost is that attention grows as sequence length squared, so long context gets expensive. The lecture closes this section by noting linear attention as one way to reduce that cost.

切

Linear attention tradeoffs 30:32

Linear attention rewrites the update so the model keeps a fixed state matrix instead of storing every token. That makes memory use constant and speeds things up, because the cost is no longer O of N squared. But it also makes long context harder. In a needle in a haystack task, the model must compress the whole passage into that state, which is why models like RWKV and Mamba can degrade more on longer sequences than standard attention.

切

Training data and post training 36:01

Training starts with a loss function. Cross entropy loss compares the model’s next-token probabilities with the ground truth token and gives lower loss when the right token gets higher probability. Pre-training data is scraped from the web, turned from HTML into plain text, filtered for language, repetition, length, and quality, and deduplicated before training. Later stages shift the mix toward target domains, then instruction fine-tuning teaches the model to answer in the desired style, and human preference training starts from paired answers.

切

Human feedback limits 45:01

Training with human feedback starts by collecting ranked answers, then fitting a reward model and using reinforcement learning to push the model toward what people seem to prefer. But the feedback is shaky. The same person may choose differently weeks later. Preferences also vary across people. That can reward flattery, since models learn to say nice things. The data itself can raise ethics issues too, including low-paid workers facing unpleasant material and poor working hours.

切

Verifiable and agent data 47:32

A cleaner route is verifiable rewards, where correctness can be checked directly. Code and math fit this well because you can test whether the answer passes cases or matches the right result. For agent training, repositories can be turned into tasks by adding bugs, reversing changes, and asking a model to fix them. Synthetic environments like this helped one model rise sharply on a verified coding benchmark. Long-context training and longer docs matter too.

切

Inference and GPU flow 53:00

Inference splits into prefill and decode. Prefill processes the whole prompt at once and is mostly compute-bound. Decode generates one token at a time and becomes memory-bound because the model must keep the key and value activations in a KV cache. Time to first token matters for prefill, and time per output token matters for decode. Speculative decoding speeds this up by having a small draft model propose tokens, then checking them against the larger model.

切

Inference time scaling 59:31

Chain-of-thought prompting makes the model reason before it answers. That means more compute at test time, and it often helps on hard tasks. New reasoning models are trained to do this more directly. They can catch mistakes, reflect, and revise themselves as they work. That costs more tokens, but it usually improves scores. Some systems now let you set a reasoning level, low, medium, or high, so you can spend more when the task needs it.

切

Sampling and context design 1:04:01

You can also sample many answers and use a verifier, or fall back on self-consistency when no verifier exists. After that, the focus shifts to agents and context engineering. The context can hold system prompts, chat history, memory, tool results, and attached files. But you cannot just stuff everything in. Long contexts get slower and less reliable, and models do worse on long-horizon retrieval tasks. The practical answer is to keep the context lean.

切

Context choices and recursion 1:13:30

Use focused context when you can, and retrieve only the most relevant files or memories instead of stuffing everything in. Add context in sequence, because inserting text at the front can break position embeddings and invalidate the KV cache. Test context formats with benchmarks, watch for compaction, and avoid it with recursive language models that can inspect and reuse the prompt.

AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.

Summarize your own YouTube video

Paste a YouTube link, article, PDF, ebook or slide deck and get a summary like this in seconds. Free to try, no sign-up needed.

⚔️ Try the YouTube summarizer

Discussion

Sign in to join the discussion. Sign in

More from the Bento Box

Browse the Bento Box →

With your OK, Microsoft Clarity and Google Analytics set cookies (data goes to the US) so we can see what breaks. Cookie details