Stanford Online

Optimizing AI Agents: Prompts vs. Weights (GEPA, GRPO) | Stanford CS329Z Lecture 9 with Michael Ryan: summary

YouTube summary12 sectionsWatch on YouTube ↗

This is an AI-generated summary of the YouTube video "Optimizing AI Agents: Prompts vs. Weights (GEPA, GRPO) | Stanford CS329Z Lecture 9 with Michael Ryan" (Stanford Online), made with Samuraize and published by Samuraize. It condenses the YouTube video into 12 titled sections you can read in a couple of minutes, each linking to the moment in the video it covers.

1
Filed under💻 Technology0 comments🍱 Add to trayReport
Study this

Optimizing AI Agents: Prompts vs. Weights (GEPA, GRPO) | Stanford CS329Z Lecture 9 with Michael Ryan

Stanford Online

切

Optimization basics 0:05

Today’s topic is optimization for AI systems. A quick poll shows how prompt choice can be surprising: for an old instruction-tuned PaLM 2L model, “take a deep breath and work on this problem step by step” wins over two other step-by-step prompts. The point is that you cannot guess which prompt will work best. You need evals to test them and see what actually improves the system.

切

Three pillars needed 4:32

An AI system needs three things before you can optimize it: data, a system, and a metric. The data can be rollouts, demonstrations, or human feedback. The system may be a language model, workflow, or agent. The metric may be pass rate, exact match, an LM judge score, cost, or latency. The lecture then turns to context optimization as last-mile adaptation, where good evals help capture intent that is hard to write down directly.

切

Entropy as prompt signal 14:30

The lecture uses a probing set to judge prompt orderings with two entropy measures. Global entropy asks how spread out the labels are across the set. Local entropy looks at the model’s token probabilities for each example. A confident answer gives low local entropy; a mixed set of options gives high local entropy. On several GPT-style models, entropy-based search found better prompt orderings than random choice, sometimes by about 8 percent.

切

Bootstrapped reasoning traces 17:31

A second line of work builds reasoning examples from the model itself. STAR generates a rationale and answer, keeps correct cases, and uses hints when the answer is wrong. DSPI and related bootstrap search do not need fine-tuning. They reject weak outputs, keep the strong ones, and then test different few-shot combinations on the dev set. This simple sampling and filtering often beats plain labeled examples and even human-written chain-of-thought demos.

切

Reflective prompt search 22:01

The next method, OPRO, treats prompt writing as a search problem. It feeds a model past instructions and their scores, then asks for a new instruction that should score higher. This produced prompts such as “take a deep breath” for math tasks, which outperformed earlier chain-of-thought wording on PaLM 2. The point is to use the model’s own feedback, often through an API, to refine the last mile of the prompt.

切

Iris classification setup 29:01

[29:01] The example task is iris classification. You give a model petal length, petal width, sepal length, and sepal width, then ask it to predict the species. The system runs a trace, scores it with exact match, and keeps only the successful ones. Correct traces become prompt examples, while wrong ones are discarded. That same process also supports instruction proposal, using the program, the data, and the successful traces to draft better prompt text.

切

Prompt search and mutation 33:31

[33:31] The collected instructions and demonstrations are then optimized together by Bayesian search. The system samples candidate prompt settings, runs them on training batches, and updates beliefs about which instructions and examples work best. A full evaluation decides the final pair to ship. After that, the lecture turns to feedback gradients, where text feedback is reflected backward through a chain of calls, and to GEPA, which keeps only Pareto-optimal prompts and creates new ones by mutation or merging.

切

System aware merge 43:31

He explains how the method takes useful pieces from different prompts and combines them into one new program. After a system-aware merge, the new candidate is checked on a mini-batch. If it improves, it is evaluated on all tasks and added back to the candidate pool. If it does not, it is discarded and the search continues until the budget runs out.

切

Privacy and teacher data 47:00

He gives a privacy example from Papillon. A private user query is rewritten by a local model into a privacy-preserving request, sent to an API model, and then combined with the original query to form the final answer. He then turns to supervised fine-tuning for agents, where a teacher solves tasks, its successful rollouts are collected, and a student model is trained on them. Strong teachers help, but a teacher that is too far ahead can be hard for a small student to learn from.

切

Rewarded agent training 58:00

Reinforcement learning for agents rolls out the student on tasks and rewards it for finishing them. Verifiable tasks matter most. Coding can use unit tests, or you can use an LLM as judge when exact checks are not possible. The example uses an expected output file, and an exact match gives reward. Two common methods are PPO and GRPO.

切

GRPO and distillation 1:03:01

PPO keeps four models in memory, trains both the policy and value model, and uses generalized advantage estimation to score each token. GRPO drops the value model and instead samples several outputs for one prompt, then compares their rewards with a z-score. That makes it lighter and more efficient. DeepSeek used repeated RL and SFT passes to build R1, and found GRPO better on math tasks. On-policy distillation then scores student moves by the teacher's log probabilities, so the student learns from its own attempts while still getting dense feedback.

切

Agent optimization choices 1:12:32

On-policy distillation uses sampled responses and a reverse KL loss, so the student learns from a teacher with per-token scores even when the reward is sparse. Qwen models use this to train smaller systems from larger ones, with higher accuracy and fewer GPU hours than reinforcement learning. The lecture then turns to agents that improve themselves through harness edits, auto research loops in repos, and the choice between prompt and weight optimization: prompts are faster and cheaper, weights help when prompts fail or the context grows too long.

AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.

Summarize your own YouTube video

Paste a YouTube link, article, PDF, ebook or slide deck and get a summary like this in seconds. Free to try, no sign-up needed.

⚔️ Try the YouTube summarizer

Discussion

Sign in to join the discussion. Sign in

More from the Bento Box

Browse the Bento Box →

With your OK, Microsoft Clarity and Google Analytics set cookies (data goes to the US) so we can see what breaks. Cookie details