DeepFloyd IF Diffusion
Setup & Prompts
Generated text embeddings using DeepFloyd IF cluster on Hugging Face. Seed=100.
All Prompts Designed:
- 'a picture of canyon'
- 'a picture of a finger'
- 'a lonely robot waiting for a bus in the snow'
- 'a painting of a piano'
- 'a painting of a zebra'
- 'a cat and a dog merged into one animal'
- 'a guitar that transforms into a bird while being played'
- 'half human half tree standing in a baseball stadium'
- 'Shohei Ohtani eating his baseball'
- 'a cyberpunk city reflected on water, pixel art style'
- 'a rainy street at night with only one yellow light on and a store opened'
- 'a moon and stars and a couple kissing and becomes two monkeys'
- 'a high quality photo'
- '' (empty prompt)
Selected 3 Prompts with num_inference_steps=20:
Same Prompts with num_inference_steps=200:
Forward Process
x_t = √(ᾱ_t)·x_0 + √(1-ᾱ_t)·ε. Applied to Campanile at t=[250, 500, 750].
Implementation: Add noise using the formula where epsilon is Gaussian noise from torch.randn_like(). The alphas_cumprod array contains cumulative product values that control scaling of the original image and noise at each timestep t.
Classical Denoising
Gaussian blur on noisy images. Removes high-freq noise but can't reconstruct edges.
Implementation: Uses torchvision.transforms.functional.gaussian_blur with varying kernel sizes. The Gaussian blur is applied directly to noisy images, but results show it's not effective for high noise levels.
One-Step Denoising
UNet predicts noise, recover x_0. Works for t≤500, struggles at t=750.
Implementation: Pass noisy image through stage_1.unet() to get noise estimate. Recover x_0 using x_0 = (x_t - √(1-ᾱ_t)·noise_pred) / √(ᾱ_t).
Iterative Denoising
DDPM sampling: t=990→0, stride=30. Iterative recovery from heavy noise.
Implementation: Loop from high timesteps to low (t=990 down to 0 with stride 30). At each step, predict noise and compute x_{t-1} from x_t using DDPM backward formula with optional random noise z.
Sampling
Generate from pure noise. Prompt: "a high quality photo"
Implementation: Start with pure Gaussian noise torch.randn(), then run the iterative denoising loop from t=max down to t=0 to generate images from scratch.





CFG
ε = ε_uncond + γ·(ε_cond - ε_uncond), γ=7. Much sharper results!
Implementation: At each timestep, run UNet twice: once with prompt embeddings (conditional) and once with empty prompt (unconditional). Combine predictions using ε = ε_uncond + γ·(ε_cond - ε_uncond).





Image-to-Image
SDEdit: partial noise + denoise
Implementation: Add noise to input image up to timestep i_start, then run denoising loop from i_start back to 0. Higher i_start = more noise = more variation from original.
SDEdit with i_start = 1, 3, 5, 7, 10, 20
1.7.1 Hand-Drawn Edits
Staff-provided avocado chair:
Custom hand-drawn images from Instagram stories: sheep, bear, stick figure, snoopy
Sheep:
Bear:
Stick Figure:
Snoopy:
1.7.2 Inpainting
Fill masked regions. At each step: denoise, then re-impose unmasked pixels.
Implementation: During denoising loop, after each UNet prediction, replace unmasked regions with original noisy values at that timestep. Masked regions are filled by the model.
1.7.3 Text-Conditional Image-to-Image
Prompt: "a lonely robot waiting for a bus in the snow". i_start = [1, 3, 5, 7, 10, 20]
Implementation: Same as SDEdit but uses different prompt embeddings during denoising. The prompt guides the transformation while preserving structure from original image.
Visual Anagrams
Dual-prompt noise averaging. The image looks different when flipped upside down!
Implementation: At each timestep, predict noise with prompt1 normally, then flip image 180°, predict noise with prompt2, flip back. Average the two noise predictions: ε = (ε1 + ε2_flipped) / 2.
Hybrid Images
Factorized diffusion with Gaussian blur (kernel=33, σ=2)
Implementation: Predict noise with prompt_low, apply Gaussian blur to get low-freq component. Predict noise with prompt_high on (original - blurred) to get high-freq. Combine: ε = ε_low_blur + ε_high_residual.
Flow Matching from Scratch
Train UNet on MNIST: single-step denoiser → time-conditioned flow matching → class-conditioned generation with CFG.
Using the UNet to Train a Denoiser
Visualize noising process: z = x + σε for σ = [0.0, 0.2, 0.4, 0.5, 0.6, 0.8, 1.0]
Implementation: Train UNet to predict clean images from noisy inputs. Add Gaussian noise ε~N(0,σ²I) to clean images, use MSELoss(predicted_clean, original_clean).
Training
Setup: D=128, batch=256, lr=1e-4, epochs=5, σ=0.5
Out-of-Distribution Testing
Test denoiser on different σ values (trained on σ=0.5 only). From top to bottom: σ = 0.0, 0.2, 0.4, 0.5, 0.6, 0.8, 1.0
Implementation: Generate noisy_images = test_images + torch.randn_like(test_images) * sigma_val for each σ, then run model(noisy_images) to test generalization.
Denoising Pure Noise
Result: Outputs blurry average of all digits (0-9). MSE loss learns the centroid—proves we need iterative denoising for generation!
Implementation: Feed pure noise torch.randn_like(images) as input with clean MNIST digits as targets. Model learns to output mean of all training images since input contains no information.
Adding Time Conditioning to UNet
Flow matching: x_t = (1-t)x_0 + tx_1, learn u(x_t,t) = x_1 - x_0. Inject t via FCBlock modulation.
Implementation: Add two FCBlock modules that modulate features at bottleneck and decoder. Time scalar t∈[0,1] reshaped to (N,1) produces modulation weights: x_unflat = unflatten(x_fc * t1) and x_dec2 = decoder(x_dec3 * t2).
Training the Time-Conditioned UNet
Setup: D=64, batch=64, lr=1e-2, epochs=10, exponential LR decay γ=0.1^(1/10)
Implementation: Compute flow matching loss using x_t = (1-t)*x_0 + t*x_1 where x_0~N(0,I) is noise, x_1 is clean image, t~Uniform(0,1). Model predicts flow vector, loss is MSELoss(predicted_flow, x_1 - x_0).
Sampling from the Time-Conditioned UNet
Euler method sampling with 50 timesteps
Implementation: Start from x_t = randn() at t=0. For each timestep i, compute t = i/num_ts, predict flow u = unet(x_t, t), update x_t = x_t + u * dt with dt = 1.0/num_ts.
Adding Class-Conditioning to UNet
One-hot class vectors (0-9) with 10% dropout for CFG training. Modulation: unflatten = c1*unflatten + t1
Implementation: Create one-hot vectors via F.one_hot(c, 10).float(). Implement dropout with mask = (torch.rand(batch_size) > 0.1).float(), then c_one_hot = c_one_hot * mask. Process through fc1_c and fc2_c FCBlocks.
Training the Class-Conditioned UNet
Setup: Same as 2.2 but with class conditioning and p_uncond=0.1
Implementation: Integrate class modulation with time modulation: x_unflat = unflatten(c1*x_fc + t1) at bottleneck and x_dec2 = decoder(c2*x_dec3 + t2) at decoder. Same flow matching loss MSELoss(predicted_flow, x_1 - x_0).
Sampling from the Class-Conditioned UNet
CFG sampling with γ=5.0. No LR scheduler: Used constant lr=5e-3, 12 epochs, gradient clipping—maintained same quality!
Implementation: At each timestep, compute two forward passes: u_cond = unet(x_t, c, t, mask=ones) and u_uncond = unet(x_t, c, t, mask=zeros). Combine: u = u_uncond + 5.0 * (u_cond - u_uncond). Use in Euler integration: x_t = x_t + u * dt.