MIT OpenCourseWareOptimizing the Full Stack for Generative Image and Video Models
MIT OpenCourseWare
Introduction and Purpose 0:00
The speaker, a research engineer at Hugging Face working on diffusion models for image and video generation, introduces a talk about optimizing the full stack for these models. He explains that the optimization challenges for generative image and video models are unusual compared to typical machine learning systems, and that the discussion will go beyond just speed and latency.
How Diffusion Models Work 3:34
Diffusion models, and their generalization called flow models, work by starting from random noise and slowly denoising it over many steps until a realistic image, video, or audio piece emerges. This denoising can be guided by text, enabling text-to-image generation. Most modern systems operate in a compressed latent space rather than directly on pixels, since pixel space diffusion is too memory and compute intensive. This requires an encoder to compress images into latent form and a decoder to convert them back, typically handled by a component called a VAE. A full text-to-video pipeline includes a text encoder, a scheduler that tracks position in the denoising process, and a diffusion network, often built as a transformer, that is invoked repeatedly before a decoder produces the final frames.
Why These Models Are Costly 11:00
Looking at a model like Flux, the diffusion transformer carries the largest memory footprint and is the most compute intensive part, unlike language models which are typically memory bound rather than compute bound. Even using efficient brain float16 precision, generating a single 1024 by 1024 image takes about 34 gigabytes of memory and roughly 7 seconds on a powerful H100 GPU without optimization. Video is worse: a 5 second, 16 FPS, 720p clip can take about 30 minutes on a state-of-the-art open video model. These long runtimes hurt interactivity, waste power, and slow down how quickly developers can iterate, making optimization essential rather than optional.
Hardware Aware Architecture Choices 15:33
Optimization should be guided by when it happens, where it happens, what is being optimized such as throughput or memory, and how it is done. Since the transformer is the most expensive part, tuning its internal shapes, like the number of attention heads or hidden dimension, to match the target hardware can noticeably boost throughput without changing the total parameter count. A paper called the efficiency misnomer shows that smaller models are not always faster or lower in compute, so efficiency needs to be judged carefully rather than assumed from model size alone.
Handling High Resolution Generation 20:31
High resolution image and video generation multiplies the dimensionality problem, since attention over very large latent representations is expensive in both memory and speed. Two common fixes are increasing the compression factor, for example moving from 8x to 16x or 24x, and recovering the resulting information loss through other techniques. The model SANA demonstrates this well, achieving a 25x reduction in generation latency for 4K images by operating on a highly compressed latent space while still recovering quality, and these architectural gains can be paired with techniques like optimized kernels or FlashAttention.
Tuning for Specific Use Cases 22:00
Beyond raw speed, models should be adapted to how they will actually be used. A model built for photorealism needs different priorities than one built for real-time interactive generation, where low latency matters more than top quality. One way to steer a model toward what users prefer is preference alignment, training on pairs of candidate images so the model learns to favor the one users like more. Models can also be extended to accept richer inputs beyond text, such as pose, segmentation maps, or Canny edge maps, giving users finer control. When further training is too costly, inference time scaling can help, searching for a better starting noise or a better prompt and checking the result against a use case specific metric like ClipScore before accepting it.
Distillation and Closing Summary 27:01
Knowledge distillation compresses a large, slow model into a smaller, faster one, and in diffusion models this often takes the form of time step distillation, which reduces the number of denoising iterations needed. Architectural compression and time step distillation can be combined for even greater gains. The speaker closes by summarizing that good optimization means designing hardware aware architectures and shapes, keeping architectures flexible enough for high resolution use cases, pairing these with latency techniques like better kernels and FlashAttention, and post training or distilling further when the specific use case or latency requirements demand it.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
