← Back to Projects

Project 5 · Diffusion & Flow Models

CS 180 / 280A · Song-Ze Yu · Seed = 100

Part A · DeepFloyd IF Part B · Flow Matching Fall 2025

DeepFloyd IF Diffusion

Part 0

Setup & Prompts

Generated text embeddings using DeepFloyd IF cluster on Hugging Face. Seed=100.

All Prompts Designed:

  • 'a picture of canyon'
  • 'a picture of a finger'
  • 'a lonely robot waiting for a bus in the snow'
  • 'a painting of a piano'
  • 'a painting of a zebra'
  • 'a cat and a dog merged into one animal'
  • 'a guitar that transforms into a bird while being played'
  • 'half human half tree standing in a baseball stadium'
  • 'Shohei Ohtani eating his baseball'
  • 'a cyberpunk city reflected on water, pixel art style'
  • 'a rainy street at night with only one yellow light on and a store opened'
  • 'a moon and stars and a couple kissing and becomes two monkeys'
  • 'a high quality photo'
  • '' (empty prompt)

Selected 3 Prompts with num_inference_steps=20:

Robot in snow
"a lonely robot waiting for a bus in the snow"
Human-tree hybrid
"half human half tree standing in a baseball stadium"
Cyberpunk city
"a cyberpunk city reflected on water, pixel art style"

Same Prompts with num_inference_steps=200:

Robot in snow (200 steps)
"a lonely robot waiting for a bus in the snow"
Human-tree hybrid (200 steps)
"half human half tree standing in a baseball stadium"
Cyberpunk city (200 steps)
"a cyberpunk city reflected on water, pixel art style"
1.1

Forward Process

x_t = √(ᾱ_t)·x_0 + √(1-ᾱ_t)·ε. Applied to Campanile at t=[250, 500, 750].

Implementation: Add noise using the formula where epsilon is Gaussian noise from torch.randn_like(). The alphas_cumprod array contains cumulative product values that control scaling of the original image and noise at each timestep t.

Original Campanile
Original Campanile
Noisy t=250
Noisy t=250
Noisy t=500
Noisy t=500
Noisy t=750
Noisy t=750
1.2

Classical Denoising

Gaussian blur on noisy images. Removes high-freq noise but can't reconstruct edges.

Implementation: Uses torchvision.transforms.functional.gaussian_blur with varying kernel sizes. The Gaussian blur is applied directly to noisy images, but results show it's not effective for high noise levels.

t=250 noisy
t=250 Noisy
t=500 noisy
t=500 Noisy
t=750 noisy
t=750 Noisy
t=250 Gaussian denoised
t=250 Gaussian denoised
t=500 Gaussian denoised
t=500 Gaussian denoised
t=750 Gaussian denoised
t=750 Gaussian denoised
1.3

One-Step Denoising

UNet predicts noise, recover x_0. Works for t≤500, struggles at t=750.

Implementation: Pass noisy image through stage_1.unet() to get noise estimate. Recover x_0 using x_0 = (x_t - √(1-ᾱ_t)·noise_pred) / √(ᾱ_t).

t=250 noisy
Noisy t=250
t=500 noisy
Noisy t=500
t=750 noisy
Noisy t=750
One-step denoised t=250
One-step denoised (t=250)
One-step denoised t=500
One-step denoised (t=500)
One-step denoised t=750
One-step denoised (t=750)
1.4

Iterative Denoising

DDPM sampling: t=990→0, stride=30. Iterative recovery from heavy noise.

Implementation: Loop from high timesteps to low (t=990 down to 0 with stride 30). At each step, predict noise and compute x_{t-1} from x_t using DDPM backward formula with optional random noise z.

Original
Original
Noisy (t=690)
Noisy (t=690)
Iterative Denoise
Iterative Denoise
One-step Denoise
One-step Denoise
Gaussian Blur
Gaussian Blur
1.5

Sampling

Generate from pure noise. Prompt: "a high quality photo"

Implementation: Start with pure Gaussian noise torch.randn(), then run the iterative denoising loop from t=max down to t=0 to generate images from scratch.

Sample 1
Sample 1
Sample 2
Sample 2
Sample 3
Sample 3
Sample 4
Sample 4
Sample 5
Sample 5
1.6

CFG

ε = ε_uncond + γ·(ε_cond - ε_uncond), γ=7. Much sharper results!

Implementation: At each timestep, run UNet twice: once with prompt embeddings (conditional) and once with empty prompt (unconditional). Combine predictions using ε = ε_uncond + γ·(ε_cond - ε_uncond).

CFG sample 1
CFG sample 1
CFG sample 2
CFG sample 2
CFG sample 3
CFG sample 3
CFG sample 4
CFG sample 4
CFG sample 5
CFG sample 5
1.7

Image-to-Image

SDEdit: partial noise + denoise

Implementation: Add noise to input image up to timestep i_start, then run denoising loop from i_start back to 0. Higher i_start = more noise = more variation from original.

SDEdit with i_start = 1, 3, 5, 7, 10, 20

i_start=1
i_start=1
i_start=3
i_start=3
i_start=5
i_start=5
i_start=7
i_start=7
i_start=10
i_start=10
i_start=20
i_start=20
1.7.1 Hand-Drawn Edits

Staff-provided avocado chair:

i_start=1
i_start=1
i_start=3
i_start=3
i_start=5
i_start=5
i_start=7
i_start=7
i_start=10
i_start=10
i_start=20
i_start=20

Custom hand-drawn images from Instagram stories: sheep, bear, stick figure, snoopy

Sheep:
Sheep i_start=1
i_start=1
Sheep i_start=3
i_start=3
Sheep i_start=5
i_start=5
Sheep i_start=7
i_start=7
Sheep i_start=10
i_start=10
Sheep i_start=20
i_start=20
Bear:
Bear i_start=1
i_start=1
Bear i_start=3
i_start=3
Bear i_start=5
i_start=5
Bear i_start=7
i_start=7
Bear i_start=10
i_start=10
Bear i_start=20
i_start=20
Stick Figure:
Stick i_start=1
i_start=1
Stick i_start=3
i_start=3
Stick i_start=5
i_start=5
Stick i_start=7
i_start=7
Stick i_start=10
i_start=10
Stick i_start=20
i_start=20
Snoopy:
Snoopy i_start=1
i_start=1
Snoopy i_start=3
i_start=3
Snoopy i_start=5
i_start=5
Snoopy i_start=7
i_start=7
Snoopy i_start=10
i_start=10
Snoopy i_start=20
i_start=20
1.7.2 Inpainting

Fill masked regions. At each step: denoise, then re-impose unmasked pixels.

Implementation: During denoising loop, after each UNet prediction, replace unmasked regions with original noisy values at that timestep. Masked regions are filled by the model.

Original
Original
Mask
Mask
Inpainted
Inpainted
1.7.3 Text-Conditional Image-to-Image

Prompt: "a lonely robot waiting for a bus in the snow". i_start = [1, 3, 5, 7, 10, 20]

Implementation: Same as SDEdit but uses different prompt embeddings during denoising. The prompt guides the transformation while preserving structure from original image.

i_start=1
i_start=1
i_start=3
i_start=3
i_start=5
i_start=5
i_start=7
i_start=7
i_start=10
i_start=10
i_start=20
i_start=20
1.8

Visual Anagrams

Dual-prompt noise averaging. The image looks different when flipped upside down!

Implementation: At each timestep, predict noise with prompt1 normally, then flip image 180°, predict noise with prompt2, flip back. Average the two noise predictions: ε = (ε1 + ε2_flipped) / 2.

Visual anagram upright
Upright: "a lonely robot waiting for a bus in the snow"
Visual anagram flipped
Flipped: "a cyberpunk city reflected on water, pixel art style"
1.9

Hybrid Images

Factorized diffusion with Gaussian blur (kernel=33, σ=2)

Implementation: Predict noise with prompt_low, apply Gaussian blur to get low-freq component. Predict noise with prompt_high on (original - blurred) to get high-freq. Combine: ε = ε_low_blur + ε_high_residual.

Hybrid image 1
Low-freq: "a picture of a finger" / High-freq: "a picture of canyon"
Hybrid image 2
Low-freq: "a painting of a piano" / High-freq: "a painting of a zebra"

Flow Matching from Scratch

Train UNet on MNIST: single-step denoiser → time-conditioned flow matching → class-conditioned generation with CFG.

1.2

Using the UNet to Train a Denoiser

Visualize noising process: z = x + σε for σ = [0.0, 0.2, 0.4, 0.5, 0.6, 0.8, 1.0]

Implementation: Train UNet to predict clean images from noisy inputs. Add Gaussian noise ε~N(0,σ²I) to clean images, use MSELoss(predicted_clean, original_clean).

Noise levels sweep
Noise levels σ sweep
1.2.1

Training

Setup: D=128, batch=256, lr=1e-4, epochs=5, σ=0.5

Training curve
Training loss curve
Training curve zoom
Loss (mid-epoch)
Training curve later
Loss (later iterations)
1.2.2

Out-of-Distribution Testing

Test denoiser on different σ values (trained on σ=0.5 only). From top to bottom: σ = 0.0, 0.2, 0.4, 0.5, 0.6, 0.8, 1.0

Implementation: Generate noisy_images = test_images + torch.randn_like(test_images) * sigma_val for each σ, then run model(noisy_images) to test generalization.

OOD sigma tests
OOD σ test results (rows: σ = 0.0 → 1.0)
1.2.3

Denoising Pure Noise

Result: Outputs blurry average of all digits (0-9). MSE loss learns the centroid—proves we need iterative denoising for generation!

Implementation: Feed pure noise torch.randn_like(images) as input with clean MNIST digits as targets. Model learns to output mean of all training images since input contains no information.

Pure noise denoising epochs
Epoch checkpoints
Pure noise comparison
Epoch 1 vs 5
Pure noise samples
Samples
2.1

Adding Time Conditioning to UNet

Flow matching: x_t = (1-t)x_0 + tx_1, learn u(x_t,t) = x_1 - x_0. Inject t via FCBlock modulation.

Implementation: Add two FCBlock modules that modulate features at bottleneck and decoder. Time scalar t∈[0,1] reshaped to (N,1) produces modulation weights: x_unflat = unflatten(x_fc * t1) and x_dec2 = decoder(x_dec3 * t2).

2.2

Training the Time-Conditioned UNet

Setup: D=64, batch=64, lr=1e-2, epochs=10, exponential LR decay γ=0.1^(1/10)

Implementation: Compute flow matching loss using x_t = (1-t)*x_0 + t*x_1 where x_0~N(0,I) is noise, x_1 is clean image, t~Uniform(0,1). Model predicts flow vector, loss is MSELoss(predicted_flow, x_1 - x_0).

Time-conditioned training
Training loss
2.3

Sampling from the Time-Conditioned UNet

Euler method sampling with 50 timesteps

Implementation: Start from x_t = randn() at t=0. For each timestep i, compute t = i/num_ts, predict flow u = unet(x_t, t), update x_t = x_t + u * dt with dt = 1.0/num_ts.

Samples overview
Sample overview
Epoch 1
Epoch 1
Epoch 5
Epoch 5
Epoch 10
Epoch 10
2.4

Adding Class-Conditioning to UNet

One-hot class vectors (0-9) with 10% dropout for CFG training. Modulation: unflatten = c1*unflatten + t1

Implementation: Create one-hot vectors via F.one_hot(c, 10).float(). Implement dropout with mask = (torch.rand(batch_size) > 0.1).float(), then c_one_hot = c_one_hot * mask. Process through fc1_c and fc2_c FCBlocks.

2.5

Training the Class-Conditioned UNet

Setup: Same as 2.2 but with class conditioning and p_uncond=0.1

Implementation: Integrate class modulation with time modulation: x_unflat = unflatten(c1*x_fc + t1) at bottleneck and x_dec2 = decoder(c2*x_dec3 + t2) at decoder. Same flow matching loss MSELoss(predicted_flow, x_1 - x_0).

Class-conditioned training
Training loss
2.6

Sampling from the Class-Conditioned UNet

CFG sampling with γ=5.0. No LR scheduler: Used constant lr=5e-3, 12 epochs, gradient clipping—maintained same quality!

Implementation: At each timestep, compute two forward passes: u_cond = unet(x_t, c, t, mask=ones) and u_uncond = unet(x_t, c, t, mask=zeros). Combine: u = u_uncond + 5.0 * (u_cond - u_uncond). Use in Euler integration: x_t = x_t + u * dt.

Epochs comparison
Epochs 1/5/10
CFG samples
CFG γ=5
Per-digit grid
Per-digit grid