Stanford OnlineStanford EE292R AI and Data for Science | Autumn 2026 | Multimodal Modeling for Precision Medicine
Stanford Online
Multimodal fusion 0:10
Multimodal modeling joins patient data from different sources. The examples here use RNA sequencing, microRNA, clinical records, histopathology images, and CT images to predict cancer prognosis or treatment response. Earlier work on large cancer cohorts showed that combining these modalities with neural networks improved results over single-data models, including in adult and pediatric brain tumors and in non-small cell lung cancer.
Missing data imaging 5:31
Clinical data are often incomplete, so the work turns to missing modalities and cross-modal modeling. RNA sequencing is valuable but not routine, while H&E slides are routine and may serve as a surrogate. A vision transformer takes whole-slide images, breaks them into 250-by-250 patches, and predicts about 25,000 genes. Trained on TCGA and GTEx and checked against spatial transcriptomics, it can impute expression and support patient grouping or drug-response prediction.
Synthetic tissue images 17:30
A cascaded diffusion model is built to generate synthetic tissue images from RNA-seq vectors. A variational autoencoder first compresses expression data to 200 means and variances. The diffusion model then starts from noise and up-samples patches to higher resolution. The team shows synthetic glioblastoma and lung adenocarcinoma images that pathologists could not tell apart from real ones. They also test cell-type distributions with HoverNet and extend the model from five tissues to 33.
Radiology embeddings 29:31
The work then turns to 3D brain MRI modeling. Autoencoders are trained on UK Biobank, neuro-oncology, ADNI, and schizophrenia data. U-Net reconstructs images best by SSIM, but the regular autoencoder gives the most useful embedding. Its features correlate with demographics, volumetric and vascular measures, genetics, and psychology, and it also performs best on survival prediction and on tasks like tumor grade, IDH1 status, Alzheimer’s subtype, and APOE.
Harmonized MRI representation 34:32
The work shifts to brain MRI, where publicly available scans from many disease groups are harmonized into about 35,000 volumes. Each image goes through the same cleanup, including bias field correction and skull stripping, and is mapped to the SRI24 atlas so the same voxel means the same brain location. A 3D masked autoencoder then learns a frozen embedding that can support prediction and synthetic image generation.
Prediction and synthetic scans 41:01
Using the learned features, simple linear probes predict IDH1 mutation status, tumor grade, brain age, and other disease markers without fine-tuning. The model also generates longitudinal MRI, conditioned on sex, disease, and technical features, to show aging and Alzheimer-like change over time. The results are still early, and the team says they still need better validation, blinded reader tests, and ways to see what the model is using.
Scoring models and transfer 53:00
A judge model scores generated images and updates the model through comparisons. The discussion then turns to data curation, public release, and using scripts to rebuild shared datasets. In practice, some models are already deployed, especially segmentation in oncology. The team also works with clinicians and has done blinded tests with pathologists.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
