Roadmap

1. An inverse question regression gets wrong
2. What a generative model learns
3. Approach 1: learn the joint, then condition it (Gaussian mixtures)
4. Approach 2: transform noise into data (flows, diffusion, flow matching)
5. Where generative models fit in machine learning
6. Generative models in science and engineering
7. Limitations and trade-offs

Anchor reading: Alves and Kitchin, Generative machine learning approaches to optimization, ChemRxiv 2026

An inverse question, the data

• ▶ one value of comes from two settings, so this inverse question has two answers

An inverse question, a question

An inverse question

• Left: least squares returns E[x | y] ≈ 0, the mean of the answers. Right: a cubic with three roots; Newton finds one
• ▶ a question with several answers needs a model that can return several answers

What a generative model learns, joint and conditional

• Joint : inputs and outputs together
• Marginal : one variable, ignoring the rest
• Conditional : fix some variables, sample the rest
• Fix outputs, sample inputs: an inverse problem
• Fix inputs, sample outputs: a forward prediction, with its spread
• Fix some of each: a question regression cannot pose
• ▶ conditioning turns one fitted distribution into a solver for forward, inverse and mixed questions

What a generative model learns, discriminative against generative

• ▶ a generative model answers more questions because it models more, including how is spread

What a generative model learns, three ways to make it learnable

1. Fit a density, then condition it: Gaussian mixtures
2. Learn a map from noise to data: flows, diffusion, flow matching
3. Chain rule, "what comes next": autoregressive models (language models)

• ▶ every generative model uses one of these three; today covers the first two

Gaussian mixture models, fitting by EM

• E-step: each point's responsibility for each component
• M-step: weights, means, covariances as responsibility-weighted averages
• Likelihood never decreases; converges to a local maximum → several starts (n_init)
• k-means with soft assignments and shaped clusters
• Choose K by BIC: data, BIC 543 (K = 1), −697 (K = 2), −673 (K = 4)
• ▶ EM fits the mixture and BIC picks ; here , one component per branch

Dempster, Laird and Rubin (1977), doi:10.1111/j.2517-6161.1977.tb01600.x; Bishop, PRML ch. 9

Gaussian mixture models, conditioning

Fix the known block at , want block :

• Second term: the Schur complement
• Mixture: condition each component , reweight by
• Result is again a mixture: exact samples, instantly
• Which columns are known is chosen after training
• ▶ conditioning is a closed-form formula, so one fitted mixture answers questions about any columns

Gaussian mixture models, live

• ▶ has two peaks at , where the regression gives one wrong number

Gaussian mixture models, root finding

• The problem: the Peng–Robinson equation of state is a cubic in the compressibility factor
• For propane at 300 K and 10 bar it has three real roots: liquid (0.0348), unstable (0.1280), vapor (0.8147)
• The recipe:
  1. sample uniformly on and compute the cubic's residual
  2. fit a GMM to the 2,000 rows
  3. condition on and sample
• ▶ a root is an input whose output is zero, so root finding is an inverse question

Gaussian mixture models, root finding results

• Top: the cubic. Below: samples of given , four models; dashed: true roots
• ▶ conditioning on gives all three roots in one set of samples

Gaussian mixture models, root finding results

• Paper, Table 1: 0.035, 0.128, 0.814 (true 0.0348, 0.1280, 0.8147)
• Our GMM, on : 0.0440, 0.1273, 0.8143 → liquid root 26% high, sensitive to fitting choices
• Same GMM on : 0.0346, 0.1257, 0.8149 → the representation is a modeling choice
• Newton's method from each cluster center → 0.0348, 0.1280, 0.8147, exact
• ▶ the generator finds every root at once; a few Newton steps make each one exact

Gaussian mixture models, optimization

• The problem: batch reactor, A → B → C. B builds up, then is consumed. How long to run for the most B?
• The recipe: sample , compute (autodiff), fit a GMM to , condition on 0

• Median 3.13 min against ; then check it is a maximum, not a minimum
• ▶ an optimum is a root of the derivative, so optimization is root finding on

Gaussian mixture models, constraints and estimation

• Constrained optimization: blend reformate, isomerate and alkylate as close as possible to a 0.4 / 0.4 / 0.2 recipe while meeting octane 87
  • add the Lagrange multipliers as columns, condition on every derivative of the Lagrangian being 0
  • samples average (0.284, 0.607, 0.109), the same blend scipy's SLSQP finds
• Parameter estimation: A → B → C in a plug-flow reactor with unknown rate constants
  • simulate many , record exit flows , fit the joint of all four
  • condition on the measured flows → a distribution of , not one fitted pair
• ▶ root finding, optimization and estimation are one recipe: build rows, fit the joint, condition on what you know

Gaussian mixture models, where they stop

• Full covariance: entries per component; EM cost
• Good for tens of variables; hopeless for a 786,432-number image
• Sharp or strongly curved structure needs many components
• ▶ Gaussian mixtures suit tens of variables; images and molecules need the transformation approach

Transforming a distribution, inverse CDF

with ; change of variables

• ▶ a generator is a map from a simple distribution to the data

Transforming a distribution, the family

• ▶ flow matching keeps the flexible ODE map and trains it by plain regression

Papamakarios et al. 2021 · Chen et al. 2018 · Ho et al. 2020 · Song et al. 2021 · Lipman et al. 2023

Flow matching, live

• ▶ with the condition as an input, the flow carries noise to both answers,

Flow matching, why the paths curve

• ▶ the field averages straight training lines, so sampling paths curve yet reach the same distribution

Flow matching, the same problems

• Same tests as the GMM (3-layer network, ~10 s on a CPU). at : −0.74 and +0.76
• Inside the data gap, (truth ±0.30): GMM −0.31 / +0.31, flow −0.41 / +0.33, no warning from either
• Peng–Robinson: vapor root found; liquid and unstable roots blurred into one band (bottom panels, root-finding figure)
• Between them (3.6% of its spread); the paper's flow, trained longer, resolves them
• ▶ flow matching solves the same problems, but it learns the conditioning, so close roots can blur

Choosing between the two

• ▶ a GMM conditions on anything but stays small; flow matching scales but answers one question

Choosing between the two, a question

Where generative models fit, the map

• Supervised: inputs → labeled outputs (L8–L12)
• Unsupervised: structure, clusters, densities; the GMM started here
• Conditional generative: trained on pairs like supervised, learns the whole
• Self-supervised: the "label" is another piece of the same data
• Next to Bayesian inversion: same kind of answer, no likelihood at inference, cost moved to training
• ▶ generative models are statistical models fitted to data, on the same map as every model since L8

Where generative models fit, still data-driven

• A generative model is a surrogate of the data: L9 and L14 apply in full
• Needs more data than regression for the same job: it also models how is distributed
• Valid only where the data are: the gap answers came back confidently
• Inherits the data's flaws: a historian model learns the plant as operated
• Answers for the surrogate: check samples with the real model
• Still has hyperparameters: K, network, steps, representation ()
• ▶ a generative model is only as good as the data it was fitted to

Where generative models fit, against the alternatives

• ▶ with a model, use a solver; use a generative model to find every solution, work from data, or seed a solver

Science and engineering, images

• GAN (generator vs discriminator) and VAE (decode a latent space): maps from simple to data
• Latent diffusion: diffusion in an autoencoder's latent space → Stable Diffusion, conditioned on text
• Stable Diffusion 3: rectified flow, i.e. flow matching at scale
• Same objective as today's 2-D figures; different scale, architecture, condition
• ▶ image generators use the training objective of today's 2-D examples, at far larger scale

Goodfellow et al. 2014 · Kingma and Welling 2013 · Rombach et al. 2022 · Esser et al. 2024

Science and engineering, molecules and proteins

• Molecular VAE: SMILES (molecules as text strings) → continuous latent space → gradient search; latent "dead areas" decode to invalid SMILES
• 3-D equivariant diffusion; DiffDock for docking poses
• RFdiffusion: hundreds of designs characterized in the lab; cryo-EM binder "nearly identical to the design model"
• AlphaFold 3: "substantially updated diffusion-based architecture"
• ▶ generated molecules and proteins count only after a validity check or a lab measurement

Gómez-Bombarelli et al. 2018 · Hoogeboom et al. 2022 · Corso et al. 2023 · Watson et al. 2023 · Abramson et al. 2024

Science and engineering, materials

• MatterGen: diffusion for crystals, fine-tuned to a target property
• 8,192 candidates per target → 4 chosen for synthesis → 1 succeeded: TaCr₂O₆, disordered
• Target bulk modulus 200 GPa; DFT 222 GPa; estimated from nanoindentation "up to 169 GPa after four measurements (158 ± 11 GPa)"
• "Within 20%" holds at 169, the top of the range; at 158 it is 21% below
• Perspective: Has generative AI solved inverse materials design?
• ▶ read the measurement behind a headline; this claim depends on which end of the range you quote

Zeni et al. 2025 · Park, Li and Walsh 2024

Science and engineering, weather and process engineering

• GenCast: conditional diffusion, with the global weather state; samples = an ensemble = UQ (L14)
• 15 days at 0.25° in ~8 min on one TPU v5; better than ECMWF ENS on 97.2% of 1,320 targets; trained 1979–2018, tested 2019
• Evaporator flowsheet (anchor paper): minimum operating cost under 12 process equations, from noisy plant-like data; best samples within 1.61% (GMM), 5.83% (flow) of SLSQP
• Invertible networks for engineering inverse problems (Ardizzone et al. 2019)
• ▶ generated samples can serve as an ensemble or a design, and they are scored against reality

Price et al. 2025 · Alves and Kitchin 2026

Science and engineering, the pattern

• Every success: the generator proposes, something else verifies
  • the lab, a held-out year of weather, DFT, the physical model
• Read the measurement behind the headline number
• Validity checks are part of the method
• ▶ plan how you will verify the samples before you train the generator

Limitations and trade-offs

• Surrogate, not system: our GMM's liquid root 0.0440 vs the true 0.0348
• Out-of-data conditioning returns confident output, no warning
• Sharp structure blurred: flow matching merged two close roots
• Validity not built in, and no single metric: check samples with the physical model
• Memorization: 94 training images extracted from 175 M Stable Diffusion samples
• Model collapse: training on generated data loses the tails, in LLMs, VAEs and GMMs
• ▶ every limitation comes from one fact: a generative model is a fitted surrogate, sampled at random

Carlini et al. 2023 · Shumailov et al. 2024

Recap

• A generative model learns the distribution; conditioning answers in any direction, every mode included
• GMM: EM to fit, Schur complement to condition, any variables, tens of dimensions
• Flow matching: regress straight-line velocities, integrate an ODE to sample, scales, fixed conditioning
• GMM: all three Peng–Robinson roots at once; Newton polishing made them exact
• Still data-driven: needs coverage, valid only near the data, answers for the surrogate
• In science: the generator proposes, a lab or a physical model verifies

Standings

Nicknames only. Everyone who skipped one still counted in every bar you saw.

Before next time

• Practice module for this session: on the course site
• Run the worked example; it needs torch and scikit-learn, both already on Colab
• Read Alves and Kitchin, §1–2, §3.1–3.5 and §3.9 (limitations)
• Notes: lectures/l16/notes.md