Multimodal Generative AI

Diffusion, flows, and the models that join images to language

About this course

Generative modelling from the distribution itself through autoencoders and VAEs, forward and reverse diffusion, sampling schedules and prediction targets, conditioning and classifier-free guidance, latent diffusion and editing, flow matching and learned transport, and the vision-language models that connect images, text and video.

What you will be able to do

  • Frame generation as learning a distribution, and compress it with an autoencoder
  • Derive the forward and reverse diffusion processes and what the network predicts
  • Control samples with conditioning, guidance and schedules
  • Compare diffusion with flow matching as transport between distributions
  • Connect images and language, and extend generation to video

What this course covers

  • Generative models, sample spaces, and data support
  • Estimation vs. sampling: the gray-mean paradox
  • Marginal distributions vs. joint distribution
  • Latent variable as shared randomness
  • Conditional probability and joint tables
  • Valid mass, support coverage, and total variation
  • Cross entropy and infinite surprise for zero-probability events
  • Conditional probability and the joint distribution table
  • Classifier-free guidance: null input and independent condition dropout
  • Score function of a Gaussian mixture and posterior mean formula
  • Classifier-free guidance scale: interpolation, extrapolation, and conventions
  • Re-weighted clean distribution: adherence–diversity tradeoff
  • Routing correctness and the limits of marginal evaluation
  • Non-commutativity of Gaussian smoothing and probability re-weighting
  • Toy contrastive setup: encoders, score matrix, and softmax probabilities
  • Gradient step on a misaligned image vector
  • Shortcut detection via controlled input intervention
  • Candidate-set effects and instance vs. semantic retrieval
  • Cosine similarity, temperature scaling, and the symmetric contrastive loss
  • Flow matching time convention and velocity vs. destination
  • Conditional path specification and supervised velocity
  • Posterior responsibilities and the marginal velocity field
  • Regression optimum and irreducible training loss
  • Continuity equation derivation of the marginal field
  • Wrong-field counter-example and diagnostic checks
  • Sources of error and simulation-free training vs. generation
  • Source representation and covariance motivation
  • One-step forward diffusion transition
  • Multi-step expansion and accumulated variance
  • Direct sampling formula and Gaussian invariance
  • Marginal distribution as a Gaussian mixture
  • Noise schedules, SNR, and finite-endpoint limits
  • Temporal coupling and two-time covariance diagnostic
  • Patch representation and spatial identity
  • Connector as trainable interface
  • Question-dependent attention routing
  • Answer head and grounded output distribution
  • Grounding versus blind-model counter-example
  • Controlled intervention experiment
  • Exhaustive enumeration and controlled evaluation
  • Source distribution and encoder-decoder setup
  • Information loss from encoder collisions
  • Two-bit encoder and exact reconstruction formula
  • Optimal decoder under squared error after a collision
  • Continuous decoder extension and non-uniqueness between codes
  • Latent sampling rule and generation quality
  • Central result: reconstruction, interpolation, and generation are distinct
  • Bayesian posterior from likelihoods and priors
  • Squared-error optimality of the posterior conditional mean
  • Score function as log-density gradient and its relation to noise prediction
  • Exact reverse conditional as a mixture and limits of Gaussian approximations
  • Linear latent codec and spatial footprints
  • Compression loss and the encoder's null space
  • Deterministic vs. stochastic recovery from a shared code
  • Latent diffusion forward corruption and conditional posterior
  • Noise-prediction / clean-estimate duality and schedule consistency
  • Decoder footprint spillover during latent editing
  • Pixel compositing guarantee and its limits
  • Euler integration error on circular flow
  • Heun's method as a predictor-corrector repair
  • Equal-query-budget comparison and convergence orders
  • Model bias as an irreducible error floor
  • Diagnostic evaluation framework for samplers
  • Noisy state parameterization and amplitude conventions
  • V coordinate, rotation, and inverse reconstruction formulas
  • Three prediction targets and their interconversion
  • Sampling update structure: deterministic vs. stochastic noise budget
  • Finite-step contraction and step-count effects on output variance
  • Central result: separating target, model, and sampling error
  • Reconstruction vs. generation: the sampling route problem
  • Marginal likelihood and exact posterior in a finite decoder
  • The Evidence Lower Bound (ELBO) and its two-term decomposition
  • Rate decomposition: mutual information plus aggregate KL
  • Gaussian encoder, KL formula, and the reparameterisation trick
  • Matched encoder–decoder family and the rate–distortion trade-off
  • Decoder sample space vs. mean image: a diagnostic for VAE blur
  • Conditional distributions over finite image sets
  • Hard versus soft conditioning
  • Conflicting conditions and the limits of compromise
  • Separate attention routes for multi-modal conditioning
  • Continuous Gaussian energy model for competing conditions
  • Exhaustive finite audit: fidelity, ablation, and diversity
  • Temporal information loss in audio representations
  • Temporal structure vs. frame-wise legality
  • Joint clip distribution vs. independent frame marginals
  • Video time vs. diffusion time as distinct indices
  • Space-time tokenisation and factorised attention
  • Endpoint ambiguity and the mean-vs-sample trade-off
  • Complementary temporal metrics and their failure modes
  • Conditional diversity, feature-map scope, and evaluation protocol

Syllabus

  1. From Data to a Distribution

    1. L1 · From Data to a Generative Distribution
    2. L2 · Autoencoders and Latent Representations
    3. L3 · Variational Autoencoders and the Cost of Compression

  2. Diffusion

    1. L4 · The Forward Diffusion Process
    2. L5 · Learning to Reverse Noise
    3. L6 · Sampling, Schedules, and Prediction Targets

  3. Conditioning, Latents and Editing

    1. L7 · Conditioning and Classifier-Free Guidance
    2. L8 · Latent Diffusion and Image Editing

  4. Flows and Transport

    1. L9 · Flow Matching and Learned Transport
    2. L10 · Sampling Flows and Comparing Generative Paths

  5. Vision, Language and Video

    1. L11 · Connecting Images and Language
    2. L12 · Vision–Language Models and Visual Grounding
    3. L13 · Controllable Generation Across Modalities
    4. L14 · Video Generation and Temporal Consistency