Probability · Geometry · Generative processes

Diffusion Exploratorium

Follow Gaussian noise all the way to a concrete shape. See the noising process, inspect the score, and understand how reverse dynamics transport probability.

.panel{display:none}#panel-foundations[hidden]{display:block!important}.tabs,.playbar{display:none}Enable JavaScript to run the experiments. The Foundations section below remains available as a reading guide.

How does Gaussian noise become a shape?

Follow the same particles from an unstructured cloud to a star. Then open the mechanism: a noisy density, its score, and a sequence of small transport steps.

A WORKED EXAMPLE · SIX STEPS

Concrete target

Reverse dynamics

Starting distribution

Seed

Preparing a reproducible particle trajectory…

600 transported samplesTracked sampleTarget contour

Generation progress 0 / 480 steps

Show 24 sample paths Target outline

Tracked sample 1

Scrub to any solver step or play the full process. A path belongs to one sample; the complete cloud is a probability distribution. The selected shape determines the score used at every step.

One trajectory, six moments

Same seed · same particle identities · same spatial scale

Turn a shape into a probability distribution

The target tells the sampler what counts as a plausible outcome.

Grey: component centres μⱼ, equally spaced along the contour. Blue: independent samples from p₀. The faint circle shows one component’s 2σ₀ radius.

Choose M = 64 points μ₁,…,μ_M along the selected outline. Give each a small Gaussian neighbourhood with σ₀ = 0.08. The resulting smooth density concentrates around a recognizable contour:

p₀(x) = 1M ∑_j=1^M 𝒩(x; μ_j, σ₀²I), x ∈ ℝ².

To sample it directly, draw a uniform component index J and z ∼ 𝒩(0,I), then set x₀ = μ_J + σ₀z. The walkthrough uses these known components to compute a score exactly.

What is being generated? Each dot is one 2D sample. Many samples trace the shape. This is a diffusion model of a contour distribution; an image diffusion model instead treats an entire pixel or latent array as one sample.

The target has nonzero thickness. A perfectly thin curve would be a singular distribution with no ordinary smooth density on ℝ² at t = 0.

Erase structure with a known noising process

Learning the reverse direction begins with a forward process we can sample exactly.

Forward time t 0.000

Signal a = 1.000Noise σ = 0.000a² + σ² = 1

This slider reuses the same x₀ and ε at each t to compare marginals smoothly. It is a coupling of noisy snapshots, not a Brownian sample path. The Diffusion lab has full forward OU paths.

Use the variance-preserving Ornstein–Uhlenbeck process

dX_t = −½X_tdt + dW_t.

Its transition from a clean point is Gaussian. An integrating factor gives X_t = e^−t/2X₀ + ∫₀ᵗe^−(t−r)/2dW_r; the integral has covariance (1−e^−t)I. Thus each noisy marginal can be sampled as

x_t = a(t)x₀ + σ(t)ε, ε ∼ 𝒩(0,I),
a(t) = e^−t/2, σ(t)² = 1−e^−t.

The signal shrinks while the Gaussian perturbation grows. Our Gaussian mixture stays analytically tractable:

p_t(x) = 1/M ∑_j 𝒩(x; aμ_j, vI),
v(t) = a²σ₀² + σ².

Keep the two variances separate: σ² is added noise conditional on x₀; v also includes the clean component’s thickness. As t → ∞, a → 0, σ → 1, and p_t approaches 𝒩(0,I).

Use the score to read the noisy distribution

A noisy observation is ambiguous. The score combines all plausible clean explanations.

Score at noise time t 0.350

Noisy density pₜScore directionsProbe xₜConditional clean mean

Probe x₁ 1.50

Probe x₂ 0.40

Drag or click the plot; arrow keys move the probe. Shading is density divided by its maximum in this view. Arrow lengths are compressed. Grey links show the four strongest component responsibilities.

The score is the spatial gradient of log density:

s_t(x) = ∇_x log p_t(x).

For this mixture, Bayes’ rule gives a responsibility for each component. Differentiating the log of the mixture yields

w_j(x,t) = exp(−‖x−aμ_j‖² / 2v)∑_ℓ exp(−‖x−aμ_ℓ‖² / 2v),
s_t(x) = [a∑_jw_jμ_j − x] / v.

The score pulls toward a weighted mean of plausible noisy components. At large t these overlap almost completely; at small t the weights become localized near the contour.

Why noise prediction learns this vector

∇_x log q(x|x₀) = −(x−ax₀)/σ² = −ε/σ,
s_t(x) = 𝔼[−ε/σ | X_t = x].

A network trained with 𝔼‖ε−ε_θ(x_t,t)‖² learns the conditional mean ε* = 𝔼[ε|x_t] at the population optimum. Hence s_θ ≈ −ε_θ/σ. This expectation, rather than the particular hidden noise realization, is what the squared-error objective identifies.

x̂₀(x,t) = 𝔼[X₀|x] = [x + σ²s_t(x)] / a.

The purple marker is this conditional clean mean. It can fall between modes, away from the contour. A one-shot conditional mean is generally not a sample from p₀; generation updates the noisy state over many noise levels. These identities use t > 0 and a > 0.

Take a small reverse step, then recompute

The sampler repeatedly asks for the score at the current location and current noise level.

In generation time u = T−t, the deterministic probability-flow velocity and stochastic reverse process are

b_ODE(x,t) = ½x + ½s_t(x),
dY_u = [½Y_u + s_t(Y_u)]du + dB_u.

The factor ½ on the ODE score is essential. The reverse SDE uses a full score and injects new noise; removing its noise without changing the drift gives the wrong density evolution.

What the numerical solver does

ODE predictor: x̃ = x + h b_ODE(x,t),
ODE corrector: x′ = x + ½h[b_ODE(x,t) + b_ODE(x̃,t−h)],
SDE update: x′ = x + h[½x+s_t(x)] + √h ξ,
ξ ∼ 𝒩(0,I), h = t−t′ > 0.

The ODE uses Heun’s method; the SDE uses Euler–Maruyama. The quadratic grid t_k = 12(1−k/480)² spends smaller steps near the narrow final density. Switch the dynamics above to see the different updates.

Read one step

Pause the main trajectory, choose a tracked sample, and press “One solver step.” Compare its drift, correction or noise, and actual displacement in the magnified view.

Local coordinates are displacements from the current sample. The zoom adapts to the step size. Blue is the drift prediction; violet is the correction (ODE) or random increment (SDE); orange is the next state.

Numerical values for the tracked sample
Quantity	Value

Understand why moving particles transports probability

The reverse process is justified at the level of densities, not by a visual resemblance.

The forward OU process satisfies the Fokker–Planck equation:

∂_tp = ½∇·(xp) + ½Δp.

Since p s_t = ∇p, this can be written as a continuity equation

∂_tp + ∇·(p v_t) = 0,
v_t(x) = −½x − ½s_t(x).

Changing clocks reverses the velocity: b_ODE = −v_t. Write ρ_u = p_T−u; then ∂_uρ + ∇·(ρb_ODE) = 0. If Φ_u is the ODE solution map,

(Φ_u)_#p_T = p_T−u,
p_T−u(Φ_u(z)) |det DΦ_u(z)| = p_T(z).

This is the mathematical meaning of transport: the flow changes locations and local volume while preserving total probability. Along a path, d log ρ_u(Y_u)/du = −∇·b_ODE.

Why the reverse SDE has the same marginals

Its drift is ½x+s_t, and its Brownian term adds ½Δρ. Substituting ρ = p_t gives

−∇·[(½x+s_t)ρ] + ½Δρ
= −½∇·(xρ) − ½Δρ = −∂_tp_t.

Thus the exact SDE and ODE evolve the same laws from p_T. Their individual paths differ; neither reconstructs the particular forward Brownian path.

Where Wasserstein geometry enters

The forward density is a W₂ gradient flow of

ℱ(ρ) = ¼∫ ‖x‖²ρ dx + ½∫ ρ log ρ dx,
∂_tρ = ∇·(ρ∇[δℱ/δρ]).

It decreases ½KL(ρ‖𝒩(0,I)), up to a constant. Running the known density curve backward recovers structure. This generative flow is not generally a W₂ shortest path. The Wasserstein lab shows displacement interpolation and JKO steps separately.

Exact statements, Gaussian initialization, and numerical error

For this finite Gaussian mixture with σ₀ > 0, densities and scores are smooth on the finite time interval. The pushforward and matching-marginal identities assume exact scores, initialization from p_T, and exact integration. The application uses a finite solver and finite samples.

The default instead starts from an exact standard Gaussian. At finite T = 12, p_T has a tiny remaining trace of the data distribution. Coupling X_T = aX₀ + σZ with Z ∼ 𝒩(0,I), independent of X₀, gives

W₂(p_T, 𝒩(0,I))² ≤ a²𝔼‖X₀‖² + 2(σ−1)².

This bounds the initial discrepancy, not its amplification by the reverse solver. “Exact noisy marginal” starts from the analytic p₁₂ and removes this discrepancy. In practical diffusion models, a learned score, a finite noise schedule, and numerical integration each contribute approximation error.

From a two-dimensional contour to an image

The structure of the algorithm stays the same; the data space and score model become much larger.

Stage	This exact, inspectable example	An image diffusion model
Represent data	x₀ ∈ ℝ²; a cloud concentrates near a contour.	x₀ ∈ ℝ^H×W×C in pixel space, or a learned latent array. One sample represents a complete image.
Add Gaussian noise	x_t = a(t)x₀ + σ(t)ε.	x_k = √ᾱ_kx₀ + √(1−ᾱ_k)ε, with independent Gaussian entries.
Estimate the noise	Compute mixture responsibilities and the exact score.	Train ε_θ(x_k,k,c) on many noisy examples; c may encode a condition such as text.
Sample repeatedly	Evaluate a time-dependent score and advance the ODE or SDE.	Apply a reverse diffusion sampler with the trained prediction at each noise level.
Read the output	Many 2D points reveal the target’s contour.	A pixel array is the image; a latent array first passes through its trained decoder.

A standard discrete reverse mean

Let α_k = 1−β_k and ᾱ_k = ∏_j≤kα_j. A common DDPM parameterization uses

μ_θ(x_k,k) = 1√α_k[x_k − β_k√(1−ᾱ_k) ε_θ(x_k,k)],
x_k−1 = μ_θ(x_k,k) + σ_k,revz.

The reverse variance is a sampler choice. One common fixed choice is β̃_k = β_k(1−ᾱ_k−1)/(1−ᾱ_k); implementations omit the final random increment. This discrete formula is explained here, while the live demo implements the continuous-time OU model.

What has actually been learned?

A trained network encodes a data prior through its denoising predictions. Noise alone does not contain a predetermined star, face, or scene. Here the target menu supplies that prior explicitly; in image models, training data and conditions determine it.

Repeated updates keep a sample compatible with a sequence of noisy distributions. Coarse organization becomes visible before fine details, but this is not simply subtracting a stored noise image or interpolating linearly to a chosen target.

This page runs the complete 2D sampling calculation locally. It contains no pretrained image network. The exact mixture makes the mechanism visible without hiding the score inside a neural model.

Check your understanding

At a large noise level, why does the score resemble −x?
Why can the conditional clean mean lie outside the target contour?
What goes wrong if you remove noise from the reverse SDE and keep its full-score drift?
Which assumptions are needed to claim (Φ_T)_#p_T = p₀?

Answers: the noisy law approaches a standard Gaussian; conditional means average competing explanations; the diffusion term disappears without the compensating half-score change; and exact scores, consistent initialization, a well-defined flow, and exact integration are required.

Mathematical framework: Ho, Jain & Abbeel, DDPM (2020); Song et al., Score-based SDEs (2021). The mixture target, live score derivation, and numerical walkthrough are original teaching implementations.

Data distribution

Dynamics

Seed

Score field

From a cloud to a contour

The score points toward higher probability. A generative process combines it with a precise time-dependent drift.

EXACT 2D SCORE · 640 PARTICLES

REVERSE SDE · NOISE → DATA

SamplesTarget centresScore direction

Process progress 0%

Drag the timeline to inspect a cached, seeded trajectory. Forward paths use exact OU transitions; reverse paths use a 360-step numerical solver. Arrows show the score direction, with compressed lengths.

6.00Forward time t

1.00Noise std. √(1−e⁻ᵗ)

—Sample RMS radius

Follow the reverse-time score

At high noise the density is nearly Gaussian. As noise falls, the score reveals the five-petal structure. New noise is still injected along the reverse SDE.

dY_u = [½Y_u + s_6−u(Y_u)] du + dB_u

Reverse clock u = 6 − t increases from 0 to 6. The + score sign follows this clock convention.

Try it in the lecture

Play the reverse SDE, then switch to the ODE with the same seed. Both recover the distribution; individual paths differ. Pause near the end to see the fine-scale score field.

What is the model actually computing?

Each target is an equally weighted mixture of 64 isotropic Gaussians with standard deviation σ₀ = 0.11. For a(t) = e^−t/2 and v(t) = a(t)²σ₀² + 1 − a(t)²:

p_t(x) = 1/M ∑_j 𝒩(x; a(t)μ_j, v(t)I)
s_t(x) = [∑_j w_j(x,t)a(t)μ_j − x] / v(t)

The weights w are normalized Gaussian responsibilities. Reverse samplers start from exact p₆, which is close to, but not exactly, a standard Gaussian. There is no neural network or training in this exhibit.

One family of densities, two ways to run it backward

The exact reverse SDE and probability-flow ODE have the same one-time marginals when initialized consistently and integrated exactly. The SDE has random paths; the ODE has deterministic paths given its initial point. Finite steps and finite samples create visible differences. Neither process retraces the forward Brownian path.

Endpoint distributions

Interpolation s 0.50

Move mass, rather than fade densities

Compare displacement interpolation with an ordinary mixture. The difference is a geometry on probability distributions.

1D QUANTILE TRANSPORT

Displacement interpolationDensity crossfadeEndpoint densities

Transport uses 600 midpoint quantiles. W₂ and the interpolated density are numerical approximations; the latter is smoothed for display. A crossfade may keep two separated peaks while transported mass occupies the space between them.

A computable JKO step: free-energy descent in Wasserstein space

For a quadratic potential V(x) = x²/2 and τ = 0.4, start from 𝒩(m,σ²) with m = 2 and σ = 0.25. Each step minimizes transport cost plus free energy. Within this Gaussian family, the minimizer is explicit.

ρ_k+1 ∈ arg min_ρ { W₂²(ρ,ρ_k)/(2h) + ℱ(ρ) }

JKO step h

Step 0m 2.000σ 0.250

Free energy (constant omitted): 2.586

Equilibrium: m = 0, σ = √0.4 ≈ 0.632. The implicit scheme approaches the continuous OU gradient flow as h → 0.

Current ρEquilibrium

Derive the update by minimizing over m and σ

For 1D Gaussians, W₂² = (m−m_k)² + (σ−σ_k)². Up to an additive constant, ℱ = (m²+σ²)/2 − τ log σ. Set both partial derivatives of the JKO objective to zero:

m_k+1 = m_k/(1+h)
σ_k+1 = [σ_k + √(σ_k² + 4hτ(1+h))] / [2(1+h)].

The positive root keeps σ > 0. Comparing the minimum with the admissible choice ρ = ρ_k shows ℱ(ρ_k+1) + W₂²(ρ_k+1,ρ_k)/(2h) ≤ ℱ(ρ_k): every exact step decreases free energy.

A mathematical spine for the lecture

Read each model at three levels: a particle process, the evolution of its density, and the objective or geometry behind that evolution.

DEFINITIONS · DERIVATIONS · REFERENCES

Notation: t is forward diffusion time; u = T − t is reverse time; τ is Langevin temperature; s_t is the score; s without a subscript is the interpolation parameter. Brownian motions B and W have covariance tI. All numerical exhibits are low-dimensional teaching models.

1 / Forward diffusion and denoising

A discrete variance-preserving chain gradually adds Gaussian noise. Let α_k = 1−β_k and ᾱ_k = ∏_j≤k α_j. Then

q(x_k|x₀) = 𝒩(√ᾱ_kx₀, (1−ᾱ_k)I).

Sample k, x₀, and ε ∼ 𝒩(0,I); form x_k = √ᾱ_kx₀ + √(1−ᾱ_k)ε. A common DDPM objective is

ℒ_simple = 𝔼 ‖ε − ε_θ(x_k,k)‖².

This noise-prediction loss is a simplified weighting of the variational training objective. At its population optimum, ε_θ estimates 𝔼[ε|x_k], giving the marginal score

s_θ(x_k,k) ≈ −ε_θ(x_k,k) / √(1−ᾱ_k).

Continuous-time exhibit: β(t)=1, ᾱ(t)=e⁻ᵗ. A single Gaussian perturbation gives the correct noisy marginal, but does not by itself specify a Brownian trajectory. [1]

2 / Reverse SDE and probability-flow ODE

For dX_t = f(X_t,t)dt + g(t)dW_t, define s_t = ∇ log p_t. Assuming sufficient smoothness and positive densities, in the increasing reverse clock u = T−t:

dY_u = [−f(Y_u,t) + g(t)²s_t(Y_u)]du
+ g(t)dB_u.

The corresponding probability-flow ODE in that same clock is

dY/du = −f(Y,t) + ½g(t)²s_t(Y).

The half factor matters. Substituting s_tp_t = ∇p_t into the Fokker–Planck equation yields an equivalent continuity equation with velocity f − ½g²s in forward time.

Exact scores and consistent initialization give matching marginals, not identical paths. Approximate scores and time discretization introduce error. These formulas assume scalar, state-independent g. [2]

3 / From Langevin particles to density evolution

dX = −∇V(X)dt + √(2τ)dB
∂_tρ = ∇·(ρ∇V) + τΔρ.

The second equation is Fokker–Planck: drift transports probability and Brownian noise spreads it. The probability flux is J = −ρ∇V − τ∇ρ. If Z = ∫e^−V/τdx is finite, π = Z⁻¹e^−V/τ has zero flux.

Euler–Maruyama gives the unadjusted Langevin update

X_n+1 = X_n − h∇V(X_n) + √(2τh)ξ_n.

Finite h generally biases the stationary law. The quartic example uses a small h for illustration, not a theorem of numerical convergence. Stability and mixing depend on the potential, step size, and initial law.

4 / Free energy as the organizing principle

ℱ(ρ) = ∫ Vρ dx + τ∫ ρ log ρ dx
ℱ(ρ) − ℱ(π) = τ KL(ρ ‖ π).

The first term favours low potential; the second favours spreading. With first variation δℱ/δρ = V + τ(logρ+1), the density equation becomes

∂_tρ = ∇·(ρ∇[δℱ/δρ]).

For smooth positive solutions with adequate decay or no-flux boundaries, integration by parts gives

dℱ/dt = −∫ ρ |∇(V+τ logρ)|² dx ≤ 0.

This is the Wasserstein gradient-flow structure. Under a globally λ-strongly convex potential (λ>0), suitable solutions satisfy W₂(ρ_t,π) ≤ e^−λtW₂(ρ₀,π). The double-well exhibit does not meet that convexity condition. [3]

5 / What geometry adds — and what it does not say

For probability measures with finite second moments, quadratic Wasserstein distance is the least root-mean-square cost of coupling their mass:

W₂²(μ,ν) = inf_{γ∈Π(μ,ν)} ∫ |x−y|² dγ(x,y).

The dynamic formulation describes the same cost through a continuity equation: minimize ∫₀¹∫ρ|v|² dx ds subject to ∂_sρ + ∇·(ρv)=0 and fixed endpoints. It connects particle velocity to motion through distribution space.

JKO replaces a small Euclidean gradient step with a minimization involving W₂²/(2h) and ℱ. The Gaussian experiment above makes this construction directly computable. Its time-discrete iterates approximate a gradient-flow curve as the step shrinks.

A diffusion-model score is an ordinary spatial gradient of log density. “Wasserstein gradient” refers to a different object: variation of a functional over distributions. The language is connected, but the two gradients are not interchangeable.

The Ricci-flow connection is a viewpoint: study an evolving geometric object through a governing flow. Here that object is a probability distribution; the simulations do not compute curvature or implement Ricci flow. [4]

Checkpoints

Checkpoint: why is “denoising = reversing noise” incomplete?

A noisy point does not uniquely identify the original datum. Generative denoising uses a distributional prior to sample plausible structure. With an exact score, reverse dynamics reconstruct the evolving laws, not the particular forward noise realization. In a learned model, the prior is encoded approximately by a trained network.

Checkpoint: can a density crossfade and a W₂ geodesic have the same mean?

Yes. Both interpolate the endpoint means linearly when those means exist. A mean alone cannot describe the distribution. Translating a narrow Gaussian illustrates the distinction: transport keeps its width, while a midpoint crossfade can be bimodal.

Checkpoint: what is the equilibrium in the Gaussian JKO experiment?

For V(x)=x²/2 and τ=0.4, π is 𝒩(0,0.4). Thus m*=0 and σ*=√0.4. Setting these values in both update formulas leaves them unchanged. Notice that 0.4 is the variance, not the standard deviation.

Primary reading

The app works without an internet connection. These links open the underlying papers when you are online.

[1] Learning to denoise

Ho, Jain & Abbeel (2020). Denoising Diffusion Probabilistic Models. Forward chains, reverse parameterizations, and noise-prediction training.

[2] Scores and stochastic dynamics

Song et al. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. Reverse SDEs and the probability-flow ODE.

[3] Free energy in Wasserstein space

Jordan, Kinderlehrer & Otto (1998). The Variational Formulation of the Fokker–Planck Equation. The time-discrete variational scheme.

[4] Dynamic optimal transport

Benamou & Brenier (2000). A computational fluid mechanics solution to the Monge–Kantorovich mass transfer problem. The kinetic-energy formulation of quadratic transport.

[5] Diffusion in a learned representation

Rombach et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. Autoencoder latents and conditional image synthesis.

Numerical methods & reproducibility

The diffusion lab uses 640 particles and a 64-component Gaussian mixture. Forward OU transitions are exact. Reverse SDE integration is Euler–Maruyama; the probability-flow ODE uses Heun's method. A quadratic reverse-time grid resolves the low-noise end. Time horizon T=6, 360 steps, 121 stored frames.

The Langevin lab has 800 particles, a fixed step h=0.005, and seed 81 on reset. Its quartic potential is not globally Lipschitz. If a nonfinite state or |x|>20 occurs, the simulation stops and reports instability; particles are not clipped to manufacture convergence. The target normalization uses numerical quadrature on [−6,6].

The transport lab uses midpoint quantiles, bisection of analytic Gaussian-mixture CDFs, and a small Gaussian display kernel. W₂ is computed from the unsmoothed quantiles. JKO statistics follow the closed-form Gaussian update.

The Noise → shape walkthrough uses 600 samples, a 64-component mixture with σ₀=0.08, horizon T=12, and 480 steps on a quadratic time grid. The default initialization is exactly N(0,I); exact p₁₂ initialization is optional. Heun and Euler–Maruyama implement the displayed ODE and SDE. The forward slider couples marginal samples with fixed noise; it is not a Brownian trajectory. The score probe evaluates the analytic density and posterior, with a rescaled density background for display. Seeds initialize a deterministic pseudorandom generator; Gaussian draws use Box–Muller. No network requests, model downloads, telemetry, or external scripts are needed.

Diffusion Exploratorium

How does Gaussian noise become a shape?

One trajectory, six moments

Turn a shape into a probability distribution

Erase structure with a known noising process

Use the score to read the noisy distribution

Why noise prediction learns this vector

Take a small reverse step, then recompute

What the numerical solver does

Understand why moving particles transports probability

Why the reverse SDE has the same marginals

Where Wasserstein geometry enters

From a two-dimensional contour to an image

A standard discrete reverse mean

What has actually been learned?

From a cloud to a contour

One family of densities, two ways to run it backward

Noise as a tool for exploration

The score connection

Move mass, rather than fade densities

A computable JKO step: free-energy descent in Wasserstein space

A mathematical spine for the lecture

1 / Forward diffusion and denoising

2 / Reverse SDE and probability-flow ODE

3 / From Langevin particles to density evolution

4 / Free energy as the organizing principle

5 / What geometry adds — and what it does not say

Checkpoints

Primary reading

[1] Learning to denoise

[2] Scores and stochastic dynamics

[3] Free energy in Wasserstein space

[4] Dynamic optimal transport

[5] Diffusion in a learned representation