Stanford OnlineStanford EE292R AI and Data for Science | Autumn 2026 | From World Models to World Agents
Stanford Online
Physical AI 0:10
LLMs and agentic AI have already changed daily life, and the next frontier is physical AI. World models are meant to bring these abilities into the physical world. Genie3 is used as the example. It generates video frames in real time from actions like arrow keys, without an explicit 3D scene.
Diffusion basics 3:03
A world model is framed as a step beyond a traditional simulator. Classic simulators keep explicit state, physics, and a rendered 3D world, which makes them interpretable and editable but limited by a sim-to-real gap. Diffusion models are introduced as the key learning method. They start from easy-to-sample Gaussian noise and learn the reverse path back to data, often in latent space with a VAE and text conditioning.
World models and actions 15:30
A world model takes the last few generated frames and conditions the next ones on actions. Those actions can be arrow keys, a text prompt, or tracked human signals such as EEG, heartbeat, gaze, hands, or head pose. In VR, that lets you touch a generated world with your real movements. A world action model goes further and predicts the future actions an agent would take, too.
Context and grounding 17:00
A video world model is a real-time, action-conditioned, causal video generator. The hard problems are long context, control, and drift, also called exposure bias. In an open loop setup, each new frame becomes part of the context, so small errors grow over time. In a closed loop setup, a robot uses real camera views and stays grounded. Maps or other persistent signals can also anchor an open loop model.
Control over video models 30:30
Camera control lets you set where the view points and where it moves from a single image. Hand control adds tracked hands so a generated world can support grabbing, petting a dog, or playing a game. A later space-time control method separates camera motion from scene time, so you can move around a moment like the bullet time scene in The Matrix.
Drift and robot policies 33:30
Autoregressive video models drift because small errors build up, leading to bad motion and color. A simple fix is backwards aggregation, or bagger: train on degraded outputs played in reverse so the model learns to recover. That same thinking leads into robotics. Instead of relying only on small labeled action data, the work reformulates policy learning around large unlabeled video and inverse dynamics.
Open questions in control 45:00
World models need 3D structure, memory of past actions, and support for planning, reasoning, and VLM integration. The talk points to several open questions: how to combine VLMs with diffusion models, how to make generation and reasoning more efficient, and how to learn representations for generative tasks. It also notes that agentic systems can use tools, verifiers, and closed loops, but safe action, global time control, growing maps, and hallucination remain hard problems for robotics.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
