Stanford OnlineStanford AA274A Principles of Robotic Autonomy | Autumn 2019 | Multi-Sensor Fusion
Stanford Online
Fusion Overview 0:05
Today’s lecture prepares you for multi-sensor perception and data fusion. It moves from the basic idea of combining camera, radar, lidar, GPS, and IMU data to the practical reasons for doing it. Each sensor has its own uncertainty. By combining them, you can reduce variance, improve object detection, and support pose estimation, which matters because you need to know where the vehicle is before it drives.
Taxonomies And Bayes 9:32
The lecture then lays out four ways to think about fusion: by the data itself, by the level at which you fuse it, by the task the sensors perform together, and by the hardware architecture. It closes this opening survey by returning to Bayesian reasoning. Prior information and new measurements are treated as random variables, and the posterior distribution gives you not just one estimate but confidence, likelihood, means, and other useful statistics.
Two sensor fusion 17:30
Two sensors measure one object at the same time, but one is more precise than the other. Their readings are treated as Gaussian measurements with different means and variances. Fusing them means multiplying the two distributions and rewriting the result so it looks like one new Gaussian. The key point is that the combined variance is smaller than either sensor alone, so the fused estimate is more accurate than the best single sensor.
Kalman assumptions 33:01
The Kalman filter is then recalled as the standard way to do fusion over time. It assumes linear dynamics with Gaussian process noise, and a linear sensor model with Gaussian measurement noise. The state update and the measurement equation both keep the Gaussian form. The main idea is the innovation sequence, the gap between measurement and model, which tells you how to balance belief in the sensor against belief in the model.
Filter assumptions 37:00
A Kalman filter works best when the model is linear, the sensor equation is linear, and the initial state belief is Gaussian with zero mean. Those three assumptions keep the posterior state estimate Gaussian, with a mean and variance you can track. When the system is not linear, you usually start with linearization, but that can fail badly if the system is very nonlinear.
Fusion setup 39:01
The filter is only a single-hypothesis method, so one Gaussian peak is fine, but multiple peaks need separate tracking. Real cars bring data association problems, since you must match each measurement to the right object. Before filtering, you also need spatial registration and temporal alignment, because lidar, radar, camera, GPS, and IMU all sit in different frames and may not share the same clock.
Kalman filter example 55:31
The estimate is built from noisy sensor data, first through prediction and then correction. Lidar and IMU values feed a two-step recursive filter, and the code is shared in MATLAB and Python. The speaker shows position, velocity, and acceleration over time against ground truth. Lidar alone gives a noisy position estimate, though the filter lowers variance. Velocity comes out better, while acceleration from the IMU is still noisy and not fully satisfying.
Adding GPS and limits 1:00:00
GPS is added as a third sensor, with its own measurement noise and sensor model row in the observation equation. The process model stays the same, but the measurement equation changes. GPS data is shown as the best of the three sensors, and differential GPS is mentioned as much more accurate, sometimes within 5 cm. The filter can still beat the best sensor on variance, but it does not remove bias. That leads to augmenting the state with bias or noise terms, using pre-whitening, and monitoring innovation for sensor or actuator faults.
Sensor fusion takeaways 1:14:00
Covariances are not just tuning knobs. Process noise is hidden in the system, but measurement uncertainty can be worked out from real sensors and used in estimation. Fusion lowers variance, yet its bigger value is robustness: sensors can fail, give wrong data, or disappear in tunnels and bad weather.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
