Stanford OnlineStanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 19: Model-Based RL
Stanford Online
Wrapping up policy optimization 0:05
This final lecture completes the course roadmap by first finishing model-free reinforcement learning and then moving into model-based reinforcement learning. Most state-of-the-art machine learning methods reduce learning to numerical optimization, the way regression problems are solved by minimizing training error step by step until convergence. Reinforcement learning does not fit neatly into this picture. Value-based methods like Q-learning are not optimizing an objective at all but doing fixed point iteration, while policy optimization methods do optimize the true reinforcement learning objective but only take one gradient step at a time before collecting new data, making them on-policy by nature. Two algorithms, trust region policy optimization (TRPO) and proximal policy optimization (PPO), were built specifically to let policies take multiple update steps before returning to the environment, closer to how ordinary optimization works.
Surrogate objectives for policy gradient 3:30
To implement policy gradients through automatic differentiation, you define a surrogate objective whose gradient equals the true policy gradient, even though the surrogate's value itself is not meaningful. One version uses the log probability of an action times the advantage; an equivalent version, called the importance sampling loss, uses the ratio between the probability of an action under a new policy and under an old policy. Both give the same gradient by the chain rule, but the importance sampling version stays well defined even when the current policy has drifted away from the one that originally collected the data, which matters once you start taking several update steps in a row.
Trust region policy optimization 8:00
The farther the policy drifts from the one that collected the data, the less accurate the advantage estimates become and the more the importance sampling ratio can distort learning. TRPO addresses this by constraining updates to a trust region, measured using KL divergence between the action distributions of the old and new policies, rather than distance in raw parameter space, since what actually matters is how much the behavior changes. TRPO worked well, especially for continuous control, but its constrained formulation required conjugate gradient methods that are tricky to implement and performed weakly with large neural networks like convolutional or transformer models.
PPO simplifies TRPO 12:00
PPO was designed to capture the same trust-region intuition without the costly constrained optimization. One version moves the KL constraint into the objective as a penalty term using Lagrange multipliers. The more popular version defines a probability ratio r(theta) between new and old policies and clips it to stay between 1 minus epsilon and 1 plus epsilon, taking the minimum between the plain ratio times advantage and this clipped version. The effect is that gradients saturate when a large step would move performance in a harmful direction, but still flow when a larger step would help, giving a simple, unconstrained way to keep updates close to the previous policy. PPO is now considered one of the most popular reinforcement learning algorithms in use.
Recap of model-free RL 18:00
Dynamic programming gave exact value estimates but required full knowledge of the system's dynamics. Monte Carlo learning replaced this with sampled returns from real interaction, unbiased but high variance and requiring a terminal state. Temporal difference learning blended sampling with bootstrapping, lowering variance and allowing online updates at the cost of bias. These ideas extend from tabular representations to function approximation with parameters theta. Value-based methods use generalized policy iteration, alternating policy evaluation and greedy improvement, as in SARSA, Q-learning, and deep Q-networks. Policy optimization methods instead learn an explicit parametric policy and directly optimize the reinforcement learning objective through estimated gradients, which push up probabilities of high-reward actions but suffer from high variance, addressed through baselines and actor-critic methods.
Introducing model-based RL recipe 27:31
Model-based reinforcement learning fits an approximate model of the dynamics from data, then uses tools from optimal control to plan through that model. The basic recipe starts with a base policy, possibly random or a given exploration policy, run in the environment to collect transition triplets of state, action, and next state. A dynamical model is then fit to this data using standard supervised learning, minimizing squared error or maximizing log likelihood, and the resulting model is used for planning. This works well when dynamics are simple, such as linear time-invariant systems, echoing earlier discussions of system identification, where the true dynamics fall within the chosen function class and only a limited number of parameters need tuning.
When Learned Models Fail to Generalize 30:30
The lecturer explains that the simple recipe of fitting a model and planning on it breaks down once dynamics are complex and nonlinear, especially when paired with high-capacity models like neural networks, because extrapolating beyond the observed data is unreliable. He illustrates this with an agent trying to move north while avoiding an obstacle. If the exploration policy only visits a limited region of the state space, the learned model will behave sensibly there but can become meaningless outside that region. Once a planning policy pushes the agent into unvisited states, this creates a distribution shift, the same covariate shift problem seen earlier in imitation learning.
Two Practical Improvements 33:31
A first simple fix is to plan in a receding horizon fashion, replanning frequently rather than trusting a long open-loop plan, similar to model predictive control. A second fix is to keep feeding newly observed transitions back into the dataset and refit the model, gradually closing the gap between the state distribution the original policy saw and the one the planning policy actually visits.
The Overfitting and Exploitation Problem 34:30
The deeper issue is that a high-capacity model needed to represent complex dynamics can overfit limited data. With only a few red data points, two very different functions can both fit them, but a wildly overfitting model will make unreliable predictions outside the observed range. An optimizer trying to maximize reward under such a model will exploit these errors, chasing regions where the model wrongly predicts high reward. The proposed remedy is to make the model express uncertainty, a distribution over possible outcomes rather than a single point estimate, so planning can reason in terms of expected reward. In the cliff example, two candidate trajectories may have the same expected endpoint, but if one has much higher uncertainty, its true expected reward is lower because some of that probability mass falls off the cliff.
Two Kinds of Uncertainty 41:31
One option is to use the entropy of the model's own output distribution, such as the standard deviation of a fitted Gaussian. But this only captures aleatoric uncertainty, the inherent noise in the process itself, not epistemic uncertainty, which is uncertainty about which model or parameters best explain the data. Two different parameter settings can fit observed data equally well yet behave very differently elsewhere, and entropy alone cannot express that kind of disagreement.
Bayesian Posteriors, Gaussian Processes, and Ensembles 45:00
The Bayesian approach models a full distribution over parameters given the data, then predicts by averaging over all plausible parameter values, called the predictive posterior distribution. A simple regression example with one data point shows a broad range of lines consistent with it, narrowing into tighter confidence bounds as more data arrives. Gaussian processes formalize this by treating data as samples from a multivariate Gaussian, where a high-dimensional sample can be unfolded into an approximate function, giving an exact analytical posterior over functions; their weakness is that computing this exactly involves matrix inversions that scale poorly with the number of data points. Bootstrap ensembles take a more empirical route, training several independent copies of a model, whose random initialization and stochastic gradient descent make them converge to different parameter values or local minima. Where these models agree, the prediction is treated as certain; where they disagree, it signals uncertainty, and averaging their predictions approximates the true posterior by combining separate Dirac-like point estimates rather than solving for an exact distribution.
Sampling Models To Score Plans 1:05:00
Given a candidate action sequence from a planning algorithm, you can use the learned posterior distribution over model parameters, theta, to judge how good that sequence really is. The approach is to sample one specific value of theta from the posterior, then propagate the dynamics forward using that particular model to produce a predicted trajectory and its reward. Repeating this sampling and propagation step many times gives a set of possible futures, and averaging their rewards approximates the expected reward under the full uncertainty of the learned model. This is a direct, concrete way of turning the earlier idea, that uncertainty lets you compute an expected reward, into something computable, though it is only one possible choice and not necessarily the best approach for every problem. When a model is already well estimated, such as a simple linear time-invariant system, this machinery may be unnecessary; it matters most when you are using high-capacity models where dynamics are hard to approximate.
PETS And Sample Efficiency Gains 1:10:00
The PETS algorithm builds on this idea by using an ensemble of neural networks to represent the posterior over the dynamics model. It samples different networks from the ensemble to generate different predicted futures, uses a cross-entropy method to generate candidate plans, scores them with these sampled models, and then acts in a model predictive control fashion, executing only the first step of the best plan before replanning. In experiments comparing PETS against model-free baselines like PPO and soft actor-critic across several tasks, the model-based method reached similar final performance but converged much faster in terms of environment interactions, illustrating the classic sample-efficiency advantage of model-based reinforcement learning. Asked for a favorite method, the instructor called model-free algorithms more mature as a technology, naming PPO and soft actor-critic as the practical go-to choices for continuous control, while noting model-based RL remains an active research area.
How The Methods Fit Together 1:14:32
The many methods covered across the course, open-loop and closed-loop, dynamic programming, reinforcement learning, imitation learning, are not competitors but pieces of a hierarchical decision-making pipeline, often combined within a single autonomy stack. Using autonomous driving as the example, the process starts from a world representation such as semantic maps, then moves to high-level goal selection, like deciding to change lanes, typically handled by closed-loop dynamic-programming-style methods because stochasticity matters most there. That goal is turned into a concrete trajectory using open-loop techniques that encode dynamics and safety constraints, and this trajectory is then tracked using model predictive control with finer-grained dynamics and safety constraints, before finally being handed to actuators through methods like PID control. Safety considerations, computed through tools such as Hamilton-Jacobi reachability, can inform both the open-loop planning and tracking stages. Learning-based, end-to-end approaches historically took hold first at the perception level and are gradually moving down this hierarchy, currently sitting somewhere between open-loop planning and trajectory tracking, but even ambitious end-to-end systems tend to leave the lowest-level control components alone, since that is where hard guarantees and fine-grained control matter most.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
