← Back to Projects

Neural Radiance Fields (NeRF)

CS 180 Project 4 · Song-Ze Yu

Part 0: Camera Calibration & 3D Scan

Calibrate camera using ArUco markers, capture 3D object scan, estimate camera poses, and prepare dataset.

Part 1: Neural Field for 2D Images

Train MLP with positional encoding to represent 2D images as continuous functions.

Part 2: Neural Radiance Field (3D)

Generate rays, sample points, and render novel views through volume rendering.

Tip: every thumbnail on this page is clickable—tap to view the original resolution.

Part 0.1: Camera Calibration

The goal of camera calibration is to estimate the intrinsic matrix and distortion coefficients of the camera. These parameters are essential for later stages (pose estimation and 3D reconstruction) and must remain constant throughout the entire capture process — this is why the assignment spec emphasizes not adjusting the camera's focal length/zoom.

By capturing multiple images of ArUco calibration tags from different angles and distances, we can solve for the camera's internal parameters using known 3D-to-2D correspondences.

ArUco Tag Calibration
▲ Sample calibration images showing ArUco tag detection from various angles and distances (33 images total)

Implementation

Note: OpenCV Version Compatibility

The spec uses syntax for OpenCV < 4.7.0. My implementation uses the updated API for newer versions. See this StackOverflow discussion for details.

▶ Camera Calibration Implementation

▶ Camera path + novel views
def create_circular_path(base_pos, num_frames):
    radius = np.sqrt(base_pos[0]**2 + base_pos[1]**2)
    fixed_z = base_pos[2]
    c2ws = []
    for i in range(num_frames):
        angle = (i / num_frames) * 2 * np.pi
        cam_x = radius * np.cos(angle)
        cam_y = radius * np.sin(angle)
        cam_pos = np.array([cam_x, cam_y, fixed_z])

        forward = -cam_pos / np.linalg.norm(cam_pos)
        world_up = np.array([0, 0, 1])
        right = np.cross(forward, world_up); right /= np.linalg.norm(right)
        up = np.cross(right, forward); up /= np.linalg.norm(up)

        c2w = np.eye(4)
        c2w[:3, 0] = right
        c2w[:3, 1] = up
        c2w[:3, 2] = forward
        c2w[:3, 3] = cam_pos
        c2ws.append(c2w)
    return c2ws

def render_novel_views(model, data_path, num_frames, save_path,
                       n_samples, near, far, device, white_bkgd=True):
    data = np.load(data_path)
    c2ws_train = data['c2ws_train']
    K = data['K'] if 'K' in data else np.array([
        [data['focal'], 0, data['images_train'].shape[2] / 2],
        [0, data['focal'], data['images_train'].shape[1] / 2],
        [0, 0, 1]
    ])
    H, W = data['images_train'].shape[1:3]

    c2ws = create_circular_path(c2ws_train[0, :3, 3], num_frames)
    eval_model = model.module if hasattr(model, 'module') else model
    eval_model.eval()

    frames = []
    with torch.no_grad():
        for c2w in c2ws:
            frame = render_image(eval_model,
                                 torch.FloatTensor(c2w),
                                 H, W, K,
                                 n_samples=n_samples,
                                 near=near, far=far,
                                 device=device)
            frames.append((np.clip(frame, 0, 1) * 255).astype(np.uint8))

    pil_frames = [Image.fromarray(f) for f in frames]
    pil_frames[0].save(save_path, save_all=True,
                       append_images=pil_frames[1:],
                       duration=33, loop=0)

Results

Images processed: 33/33 images (100% detection rate)

Total tag detections: 198 tags

Reprojection Error: 0.786 pixels

Camera Intrinsic Matrix (K):

[[957.34    0.00  795.40]
 [  0.00  949.21  594.72]
 [  0.00    0.00    1.00]]

Distortion Coefficients: [k₁, k₂, p₁, p₂, k₃]

[0.261, -1.723, 0.018, 0.027, 2.748]

Part 0.2: Capturing a 3D Object Scan

I placed my Miffy doll next to the calibration tag and shot 38 photos while orbiting the bunny, keeping exposure/focus fixed so the poses stay consistent. The raw captures live at images/0-miffy-two/. (Fun fact: I first tried the CalHacks 12.0 bear, but the busy background made training unstable, so I re-shot with Miffy—Part 1 still uses the bear!)

Miffy and CalBear comparison
▲ Left: new Miffy capture used for Part 2. ▲ Right: original CalHacks bear from Part 1.

Before training Part 2, I resize the photos down to 373×280 so they match the intrinsics of the provided Lego dataset. A tiny Pillow script batches the conversion:

▼ Batch resizing script
import os
from PIL import Image

input_folder = "images/0-miffy-two"
output_folder = "images/0-miffy-two-resize"
target_size = (373, 280)

os.makedirs(output_folder, exist_ok=True)

for filename in os.listdir(input_folder):
    if filename.lower().endswith((".jpg", ".jpeg", ".png")):
        img_path = os.path.join(input_folder, filename)
        img = Image.open(img_path)
        img_resized = img.resize(target_size, Image.LANCZOS)
        img_resized.save(os.path.join(output_folder, filename))
        print(f"Done: {filename}")

print("All images resized!")

The resized set in images/0-miffy-two-resize/ becomes the dataset I feed into the NeRF dataloader later on.

Part 0.3: Camera Pose Estimation

The goal of this step is to estimate the extrinsic parameters (rotation and translation) for each camera pose. This is known as the Perspective-n-Point (PnP) problem: given a set of 3D points in world coordinates and their corresponding 2D projections in an image, we solve for the camera's position and orientation in 3D space.

Implementation

Important: World-to-Camera vs Camera-to-World

OpenCV's solvePnP() returns the world-to-camera (w2c) transformation.
For NeRF, we need the camera-to-world (c2w) transformation, so we must invert the result:

c2w = [R | t] = [R₀₀  R₀₁  R₀₂  tₓ]
      [0 | 1]   [R₁₀  R₁₁  R₁₂  tᵧ]
                [R₂₀  R₂₁  R₂₂  t_z]
                [ 0    0    0    1 ]

▶ Pose Estimation Implementation

Visualization with Viser

To verify our pose estimation, we visualize all camera frustums in 3D space using viser.

▶ 3D Frustum Visualization

Results

Camera Frustums View 1
▲ 3D visualization of camera frustums (view 1)
Camera Frustums View 2
▲ 3D visualization of camera frustums (view 2)

Part 0.4: Dataset Preparation

The final step is to undistort all images and package everything into a dataset format ready for NeRF training. This combines all previous steps from Part 0.1 to 0.3.

Implementation

▶ Dataset Creation Implementation

Dataset Structure

Output: my_data.npz

The packaged dataset contains:

  • images_train: Training images (undistorted RGB)
  • c2ws_train: Camera-to-world matrices for training (0.7)
  • images_val: Validation images
  • c2ws_val: Camera-to-world matrices for validation (0.15)
  • c2ws_test: Camera-to-world matrices for test (0.15)
  • K: Full intrinsic matrix (switched from scalar focal because it made Part 2.6 result better)

Part 1: Neural Field for 2D Images

Before jumping into 3D NeRF, which learns the mapping F: {x, y, z, θ, φ} → {r, g, b, σ}, we start with 2D as a simpler introduction. In 2D, we only need to solve F: {u, v} → {r, g, b}, where {u, v} represents 2D pixel coordinates.

1.1 Sinusoidal Positional Encoding (PE)


Positional encoding helps neural networks capture high-frequency details by transforming input coordinates into a higher-dimensional space using sinusoidal functions. Without PE, MLPs struggle to represent fine details in images.

The encoding applies sine and cosine functions at different frequencies:

2D Neural Field Architecture
▼ Positional Encoding Implementation
class PositionalEncoding(nn.Module):
    def __init__(self, L=10):
        super().__init__()
        self.L = L

    def forward(self, x):
        out = [x]
        for i in range(self.L):
            out.append(torch.sin(2**i * np.pi * x))
            out.append(torch.cos(2**i * np.pi * x))
        return torch.cat(out, dim=-1)

For L=10 frequency bands, a 2D coordinate is transformed from dimension 2 to dimension 2 + 2×L×2 = 42.

1.2 Multilayer Perceptron (MLP) Architecture

Our neural field network follows the architecture shown below:

2D Neural Field Architecture
▲ Network architecture: 2D coordinates → PE → 3 hidden layers (256 units) → RGB output

The network consists of:

  • Input: 2D pixel coordinates (u, v)
  • Positional Encoding: Expands input to 42 dimensions
  • Hidden Layers: 3 fully-connected layers with 256 units each, using ReLU activation
  • Output Layer: Maps to RGB values (3 channels) with Sigmoid activation to ensure [0,1] range
▼ Neural Field 2D Implementation
class NeuralField2D(nn.Module):
    def __init__(self, L=10, hidden_dim=256):
        super().__init__()
        self.pe = PositionalEncoding(L)
        pe_dim = 2 + 2 * L * 2  # 42 dimensions

        self.mlp = nn.Sequential(
            nn.Linear(pe_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 3),
            nn.Sigmoid()
        )

    def forward(self, coords):
        encoded = self.pe(coords)
        return self.mlp(encoded)

1.3 Pixel Dataloader

Since high-resolution images are memory-intensive, we implement a dataloader that randomly samples N pixels at every iteration for training. This approach:

  • Returns N×2 normalized 2D coordinates in [0,1]
  • Returns corresponding N×3 RGB color values in [0,1]
  • Enables efficient mini-batch training without loading entire images
▼ Pixel Dataloader Implementation
class PixelDataloader:
    def __init__(self, img, batch_size=10000):
        self.img = img / 255.0  # Normalize to [0,1]
        self.H, self.W = img.shape[:2]
        self.batch_size = batch_size

        # Create normalized coordinate grid
        y, x = np.meshgrid(np.arange(self.H), np.arange(self.W), indexing='ij')
        self.coords = np.stack([x / self.W, y / self.H], axis=-1).reshape(-1, 2)
        self.colors = self.img.reshape(-1, 3)

    def sample(self):
        """Randomly sample batch_size pixels"""
        idx = np.random.choice(len(self.coords), self.batch_size, replace=False)
        coords = torch.FloatTensor(self.coords[idx])
        colors = torch.FloatTensor(self.colors[idx])
        return coords, colors

    def get_all(self):
        """Get all pixels (for validation/visualization)"""
        coords = torch.FloatTensor(self.coords)
        colors = torch.FloatTensor(self.colors)
        return coords, colors

1.4 Training: Loss Function, Optimizer, and Metric

We train the neural field to minimize the difference between predicted and ground truth pixel colors using:

  • Loss Function: Mean Squared Error (MSE) between predicted and actual RGB values
  • Metric: Peak Signal-to-Noise Ratio (PSNR) = 10 × log₁₀(1/MSE)
  • Optimizer: Adam with learning rate 1e-2
  • Training: 1000-3000 iterations with batch size of 10,000 pixels
▼ Training Loop Implementation
def train_neural_field(img_path, L=10, hidden_dim=256, lr=1e-2, iters=3000, batch_size=10000, save_dir='images'):
    os.makedirs(save_dir, exist_ok=True)

    device = torch.device('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu')
    print(f'Using device: {device}')

    img = skio.imread(img_path)
    if img.ndim == 2:
        img = np.stack([img] * 3, axis=-1)

    H, W = img.shape[:2]

    model = NeuralField2D(L=L, hidden_dim=hidden_dim).to(device)
    dataloader = PixelDataloader(img, N=batch_size)
    optimizer = torch.optim.Adam(model.parameters(), lr=lr)
    criterion = nn.MSELoss()

    psnr_history = []
    save_iters = [0, 100, 500, 1000, 2000, iters-1]

    for it in range(iters):
        coords, colors = dataloader.sample()
        coords, colors = coords.to(device), colors.to(device)

        pred_colors = model(coords)
        loss = criterion(pred_colors, colors)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        psnr = psnr_from_mse(loss)
        psnr_history.append(psnr.item())

        if it % 100 == 0 or it == iters - 1:
            print(f'Iter {it:4d}: Loss={loss.item():.6f}, PSNR={psnr.item():.2f} dB')

        # Save visualization at key iterations
        if it in save_iters:
            with torch.no_grad():
                y, x = np.meshgrid(np.arange(H), np.arange(W), indexing='ij')
                test_coords = np.stack([x / W, y / H], axis=-1).reshape(-1, 2)
                test_coords = torch.FloatTensor(test_coords).to(device)

                pred = []
                for i in range(0, len(test_coords), batch_size):
                    pred.append(model(test_coords[i:i+batch_size]).cpu())
                pred = torch.cat(pred, dim=0)

                pred_img = pred.reshape(H, W, 3).numpy()

                # Save comparison: GT, Prediction, Error
                plt.figure(figsize=(12, 4))
                plt.subplot(1, 3, 1)
                plt.imshow(img / 255.0)
                plt.title('Ground Truth')
                plt.axis('off')

                plt.subplot(1, 3, 2)
                plt.imshow(pred_img)
                plt.title(f'Prediction (iter {it})')
                plt.axis('off')

                plt.subplot(1, 3, 3)
                plt.imshow(np.abs(img / 255.0 - pred_img))
                plt.title('Error')
                plt.axis('off')

                plt.tight_layout()
                plt.savefig(f'{save_dir}/iter_{it:04d}.png', bbox_inches='tight', dpi=150)
                plt.close()

    # Save PSNR curve
    plt.figure(figsize=(10, 6))
    plt.plot(psnr_history)
    plt.xlabel('Iteration')
    plt.ylabel('PSNR (dB)')
    plt.title('Training PSNR')
    plt.grid(True)
    plt.savefig(f'{save_dir}/psnr_curve.png', bbox_inches='tight', dpi=150)
    plt.close()

    return model, psnr_history

1.5 Results and Analysis

1) Model Architecture

  • Positional Encoding: L = 10 frequency bands (input dimension: 42)
  • Network Depth: 3 hidden layers
  • Layer Width: 256 units per hidden layer
  • Activation: ReLU (hidden layers), Sigmoid (output layer)
  • Learning Rate: 1e-2 (Adam optimizer)
  • Batch Size: 10,000 pixels per iteration
  • Training Iterations: 3,000

2) Training Progression

The following grids show how the neural field sharpens each image across training iterations.

Test Image (Fox)

Fox - Iteration 0
▲ Iter 0
Fox - Iteration 100
▲ Iter 100
Fox - Iteration 500
▲ Iter 500
Fox - Iteration 1000
▲ Iter 1000
Fox - Iteration 2000
▲ Iter 2000
Fox - Iteration 2999
▲ Iter 2999

My Own Image (CalHacks Bear)

CalBear - Iteration 0
▲ Iter 0
CalBear - Iteration 100
▲ Iter 100
CalBear - Iteration 500
▲ Iter 500
CalBear - Iteration 1000
▲ Iter 1000
CalBear - Iteration 2000
▲ Iter 2000
CalBear - Iteration 2999
▲ Iter 2999

3) Hyperparameter Tuning

We experiment with different values of L and layer width to understand their impact on reconstruction quality.
Using very low values reveals how these hyperparameters affect the model's ability to capture fine details.

Fox - Hyperparameter Grid
Fox: effect of L and layer width.
CalBear - Hyperparameter Grid
CalBear: same search space, similar trends.

Observations

  • Positional Encoding (L): Lower L values struggle to capture high-frequency details, resulting in blurry outputs. Higher L values enable the network to represent fine textures and edges.
  • Layer Width: Narrower networks (fewer neurons per layer) have limited capacity, leading to smoother, less detailed reconstructions. Wider networks can fit more complex patterns.

4) Training PSNR Curves

This shows reconstruction quality improvement during training. Higher PSNR indicates better image quality.

Fox PSNR Curve
▲ Fox training PSNR progression
CalBear PSNR Curve
▲ CalHacks bear training PSNR progression

The __main__ block above performs a quick sanity sweep: verifying UV indexing, sampling random rays, and ensuring sample_along_rays still produces reasonable shapes before plugging the dataloader into training.

Part 2.1: Ray Generation

This step turns every 2D pixel into a 3D ray—that’s the data the NeRF will actually learn from. For each pixel we recover its camera-space direction, rotate/translate it into world space using the pose, and treat the camera center plus that direction as a ray ready for sampling.

Coordinate Conventions

  • Intrinsic matrix K: Encodes the focal lengths f_x, f_y and principal point (o_x, o_y). I assume square pixels so f_x = f_y = focal when using the provided synthetic datasets.
  • Extrinsics: I store poses as camera-to-world (c2w) matrices, which is the inverse of OpenCV’s w2c output. Each pose is [R | t] where t is the camera origin expressed in world space.
  • Pixel centers: Following the spec, I add 0.5 to each integer pixel index so that rays pass through pixel centers when forming the meshgrid.
Mapping a pixel to a 3D ray
▲ Camera-centric view of the mappings implemented below: pixels → camera frame → world rays.

Implementation

The code mirrors the three conceptual blocks above. First, pixel_to_camera undoes the projection equation s · [u, v, 1]^⊤ = K · [x_c, y_c, z_c]^⊤. Then pixel_to_ray rotates and translates those points into world space and subtracts the camera origin to obtain directions. Finally, get_rays packages everything for a full-resolution image by creating a dense grid of pixel centers.

▼ Ray Generation Functions (PyTorch)
def pixel_to_camera(K, uv, s, device='cpu'):
    if not isinstance(K, torch.Tensor):
        K = torch.FloatTensor(K).to(device)
    if not isinstance(uv, torch.Tensor):
        uv = torch.FloatTensor(uv).to(device)

    u, v = uv[:, 0], uv[:, 1]
    f_x, f_y = K[0, 0], K[1, 1]
    o_x, o_y = K[0, 2], K[1, 2]

    x_c = (u - o_x) / f_x * s
    y_c = (v - o_y) / f_y * s
    z_c = torch.ones_like(u) * s
    return torch.stack([x_c, y_c, z_c], dim=-1)

def pixel_to_ray(K, c2w, uv):
    if not isinstance(c2w, torch.Tensor):
        c2w = torch.tensor(c2w, dtype=torch.float32)

    device = c2w.device
    x_c = pixel_to_camera(K, uv, s=1.0, device=device)
    R, t = c2w[:3, :3], c2w[:3, 3]

    x_w = (R @ x_c.T).T + t.unsqueeze(0)
    r_o = t.unsqueeze(0).expand_as(x_w)
    r_d = x_w - r_o
    r_d = r_d / torch.norm(r_d, dim=-1, keepdim=True)
    return r_o, r_d

def get_rays(H, W, focal, c2w):
    device = c2w.device if isinstance(c2w, torch.Tensor) else 'cpu'
    K = torch.FloatTensor([
        [focal, 0, W / 2],
        [0, focal, H / 2],
        [0, 0, 1]
    ]).to(device)

    i, j = torch.meshgrid(
        torch.arange(H, device=device),
        torch.arange(W, device=device),
        indexing='ij'
    )
    uv = torch.stack([j + 0.5, i + 0.5], dim=-1).reshape(-1, 2).float()
    return pixel_to_ray(K, c2w, uv)

Part 2.2: Point Sampling

Now that every pixel is a ray, we have to choose a subset of rays per batch and discretize each ray into 3D samples for the NeRF.

Sampling Rays Across Images

I reuse the Part 1 pixel sampler but extend it across all views: pick M images, draw N/M pixels from each, and convert them to rays. Adding +0.5 keeps rays centered on the pixels.

n_rays = 1024
select_inds = np.random.choice(len(rays_o), size=n_rays, replace=False)
rays_o = rays_o[select_inds]
rays_d = rays_d[select_inds]

pts, z_vals = sample_along_rays(rays_o, rays_d, n_samples=64)

print(f'Points shape: {pts.shape}')
print(f'Z values shape: {z_vals.shape}')
print(f'Z values range: [{z_vals.min():.2f}, {z_vals.max():.2f}]')
print(f'First point: {pts[0, 0]}')

Sampling Points Along Rays

Each ray is continuous, so I slice it into n_samples depths between near=2.0 and far=6.0. During training I jitter inside each interval (stratified sampling) so the network doesn’t overfit to a single lattice.

▼ Stratified Sampling Along Rays
def sample_along_rays(rays_o, rays_d, near=2.0, far=6.0,
                      n_samples=64, perturb=True):
    N = rays_o.shape[0]

    # Linear samples in normalized depth, then map to [near, far]
    t_vals = torch.linspace(0., 1., n_samples, device=rays_o.device)
    z_vals = near * (1. - t_vals) + far * t_vals          # (n_samples,)
    z_vals = z_vals.expand(N, n_samples)                  # (N, n_samples)

    if perturb:
        mids = 0.5 * (z_vals[:, :-1] + z_vals[:, 1:])
        upper = torch.cat([mids, z_vals[:, -1:]], dim=-1)
        lower = torch.cat([z_vals[:, :1], mids], dim=-1)
        t_rand = torch.rand_like(z_vals)
        z_vals = lower + (upper - lower) * t_rand

    pts = rays_o.unsqueeze(1) + rays_d.unsqueeze(1) * z_vals.unsqueeze(2)
    return pts, z_vals

The function returns both the sampled 3D coordinates and their depths. Later, the renderer uses z_vals to compute alpha weights and volume accumulation. I typically choose n_samples = 64 for training and disable perturbation at evaluation time for deterministic renders.

Part 2.3: Data Loading

This section is basically Part 1’s pixel sampler leveled up: now every batch needs to return ray origins, ray directions, and the associated RGB colors across all camera views.

Multiview Rays Dataloader

I precompute the rays for each image once, flatten everything, and then sample by indexing into those big tensors. This keeps the per-iteration cost tiny and makes it easy to sanity check correspondence between UV coordinates and RGB values.

▼ RaysData + NerfDataloader (high level)
import torch
import numpy as np
from nerf_part2_1_rays import get_rays

class RaysData:
    def __init__(self, images, K, c2ws, device='cpu'):
        if not isinstance(images, torch.Tensor):
            images = torch.FloatTensor(images)
        if not isinstance(c2ws, torch.Tensor):
            c2ws = torch.FloatTensor(c2ws)

        self.images = images.to(device)
        self.c2ws = c2ws.to(device)
        self.device = device
        N, H, W, _ = self.images.shape

        all_rays_o, all_rays_d, all_pixels, all_uvs = [], [], [], []

        for i in range(N):
            rays_o, rays_d = get_rays(H, W, K, self.c2ws[i])
            all_rays_o.append(rays_o)
            all_rays_d.append(rays_d)
            all_pixels.append(self.images[i].reshape(-1, 3))

            y, x = torch.meshgrid(torch.arange(H, device=device), torch.arange(W, device=device), indexing='ij')
            all_uvs.append(torch.stack([x, y], dim=-1).reshape(-1, 2).long())

        self.rays_o = torch.cat(all_rays_o, dim=0)
        self.rays_d = torch.cat(all_rays_d, dim=0)
        self.pixels = torch.cat(all_pixels, dim=0)
        self.uvs = torch.cat(all_uvs, dim=0)

    def sample_rays(self, n):
        indices = torch.randint(0, len(self.rays_o), (n,), device=self.device)
        return self.rays_o[indices], self.rays_d[indices], self.pixels[indices]

class NeRFDataloader:
    def __init__(self, images, c2ws, K_or_focal, H, W, batch_size, device='cpu'):
        if isinstance(K_or_focal, np.ndarray):
            K = K_or_focal
        else:
            focal = float(K_or_focal)
            K = np.array([[focal, 0, W/2], [0, focal, H/2], [0, 0, 1]])
        self.rays_data = RaysData(images, K, c2ws, device)
        self.batch_size = batch_size

    def sample(self):
        return self.rays_data.sample_rays(self.batch_size)

Validation

I used the provided plotter to draw camera frustums and sampled rays. Sampling across all views produces a spiky hedgehog, while sampling from a single pose keeps the rays nicely inside one frustum—perfect for catching any indexing bug.

Random rays from all views
▲ Random rays from multiple cameras.
Rays from first camera
▲ Rays sampled only from the first camera.

Part 2.4: NeRF Network

The network now needs to map 3D locations and view directions to both color and density. Compared to Part 1’s tiny MLP, this model is deeper, runs on 3D coordinates, and conditions its RGB prediction on the viewing direction.

NeRF MLP architecture
▲ Architecture from the spec
▼ NeRF model (PyTorch)
import torch.nn as nn

class PositionalEncoding(nn.Module):
    def __init__(self, L=10):
        super().__init__()
        self.L = L

    def forward(self, x):
        out = [x]
        for i in range(self.L):
            freq = 2 ** i * np.pi
            out.extend([torch.sin(freq * x), torch.cos(freq * x)])
        return torch.cat(out, dim=-1)

class NeRF(nn.Module):
    def __init__(self, L_pos=10, L_dir=4, hidden_dim=256):
        super().__init__()
        self.pe_pos = PositionalEncoding(L_pos)
        self.pe_dir = PositionalEncoding(L_dir)

        pos_dim = 3 + 3 * 2 * L_pos
        dir_dim = 3 + 3 * 2 * L_dir

        self.lin1 = nn.Linear(pos_dim, hidden_dim)
        self.lin2 = nn.Linear(hidden_dim, hidden_dim)
        self.lin3 = nn.Linear(hidden_dim, hidden_dim)
        self.lin4 = nn.Linear(hidden_dim, hidden_dim)

        self.lin5 = nn.Linear(hidden_dim + pos_dim, hidden_dim)
        self.lin6 = nn.Linear(hidden_dim, hidden_dim)
        self.lin7 = nn.Linear(hidden_dim, hidden_dim)
        self.lin8 = nn.Linear(hidden_dim, hidden_dim)

        self.density_lin = nn.Linear(hidden_dim, 1)
        self.feature_lin = nn.Linear(hidden_dim, hidden_dim)
        self.color_lin1 = nn.Linear(hidden_dim + dir_dim, 128)
        self.color_lin2 = nn.Linear(128, 3)

        nn.init.constant_(self.density_lin.bias, 0.1)

    def forward(self, pts, dirs):
        pts_enc = self.pe_pos(pts)
        dirs_enc = self.pe_dir(dirs)

        x = torch.relu(self.lin1(pts_enc))
        x = torch.relu(self.lin2(x))
        x = torch.relu(self.lin3(x))
        x = torch.relu(self.lin4(x))
        x = torch.cat([x, pts_enc], dim=-1)
        x = torch.relu(self.lin5(x))
        x = torch.relu(self.lin6(x))
        x = torch.relu(self.lin7(x))
        x = torch.relu(self.lin8(x))

        sigma = torch.relu(self.density_lin(x))
        feat = self.feature_lin(x)
        feat_dir = torch.cat([feat, dirs_enc], dim=-1)
        rgb = torch.relu(self.color_lin1(feat_dir))
        rgb = torch.sigmoid(self.color_lin2(rgb))
        return rgb, sigma

model = NeRF(L_pos=10, L_dir=4, hidden_dim=256)

Part 2.5: Volume Rendering

This is where the neural outputs turn into actual pixels. Given densities σ, colors c, and depths z sampled along every ray, we integrate them with the discrete volume-rendering equation from the handout:

Volume rendering equation
▲ Discrete approximation of the continuous volume-rendering integral used in NeRF.
▼ Volume rendering helpers
import torch
import numpy as np

def volrend(sigmas, rgbs, step_size):
    N, n_samples = sigmas.shape[:2]
    dists = torch.full((N, n_samples), step_size, device=sigmas.device)
    alpha = 1.0 - torch.exp(-sigmas.squeeze(-1) * dists)
    trans = torch.cumprod(
        torch.cat([torch.ones_like(alpha[:, :1]), 1.0 - alpha[:, :-1]], dim=-1),
        dim=-1
    )
    weights = alpha * trans
    return torch.sum(weights.unsqueeze(-1) * rgbs, dim=1)

def volume_rendering(rgb, sigma, z_vals, rays_d, white_bkgd=False):
    dists = z_vals[:, 1:] - z_vals[:, :-1]
    dists = torch.cat([dists, torch.full_like(dists[:, :1], 1e10)], dim=-1)
    dists = dists * torch.norm(rays_d, dim=-1, keepdim=True)

    alpha = 1.0 - torch.exp(-sigma.squeeze(-1) * dists)
    trans = torch.cumprod(
        torch.cat([torch.ones_like(alpha[:, :1]), 1.0 - alpha[:, :-1]], dim=-1),
        dim=-1
    )
    weights = alpha * trans

    rgb_map = torch.sum(weights.unsqueeze(-1) * rgb, dim=1)
    depth_map = torch.sum(weights * z_vals, dim=-1)
    acc_map = torch.sum(weights, dim=-1)

    if white_bkgd:
        rgb_map = rgb_map + (1.0 - acc_map.unsqueeze(-1))
    return rgb_map, depth_map, acc_map

Run Part 2 on the Lego Dataset

I packaged all subcomponents into a single script that reproduces the NeRF paper’s Lego experiment. It handles argument parsing, ray sampling, training, validation renders, and novel-view generation.

▼ CLI + imports
import os
import sys

if __name__ == '__main__':
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('--dataset', type=str, default='lego',
                        choices=['lego', 'lafufu', 'mydata', 'rabbit', 'miffy', 'all'])
    parser.add_argument('--multi_gpu', action='store_true')
    parser.add_argument('--gpu', type=str, default=None,
                        help='GPU device ID (e.g., "0" or "0,1,2")')
    args = parser.parse_args()

    if args.gpu is not None:
        os.environ['CUDA_VISIBLE_DEVICES'] = args.gpu
        print(f'Using GPU(s): {args.gpu}')
else:
    args = None

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
from tqdm import tqdm

from nerf_part2_1_rays import get_rays
from nerf_part2_2_sampling import sample_along_rays
from nerf_part2_3_dataloader import NeRFDataloader
from nerf_part2_4_network import NeRF
from nerf_part2_5_rendering import volume_rendering
▶ Rendering helpers
def render_rays(model, rays_o, rays_d, n_samples, near, far,
               device, white_bkgd=True, perturb=False):
    pts, z_vals = sample_along_rays(rays_o, rays_d, near, far,
                                    n_samples, perturb=perturb)

    N, S = pts.shape[:2]
    pts_flat = pts.reshape(-1, 3)
    dirs_flat = rays_d.unsqueeze(1).expand(-1, S, -1).reshape(-1, 3)

    rgb, sigma = model(pts_flat, dirs_flat)
    rgb = rgb.reshape(N, S, 3)
    sigma = sigma.reshape(N, S, 1)

    return volume_rendering(rgb, sigma, z_vals, rays_d,
                            white_bkgd=white_bkgd)

def render_image(model, c2w, H, W, K_or_focal, chunk=1024,
                 n_samples=64, near=2.0, far=6.0, device='cpu',
                 white_bkgd=True):
    rays_o, rays_d = get_rays(H, W, K_or_focal, c2w)
    rays_o, rays_d = rays_o.to(device), rays_d.to(device)

    rgb_chunks = []
    for i in range(0, len(rays_o), chunk):
        rgb, _, _ = render_rays(model,
                                rays_o[i:i+chunk],
                                rays_d[i:i+chunk],
                                n_samples, near, far,
                                device=device, white_bkgd=white_bkgd,
                                perturb=False)
        rgb_chunks.append(rgb.cpu())

    rgb_map = torch.cat(rgb_chunks, dim=0).reshape(H, W, 3)
    return rgb_map.numpy()
▶ Training + evaluation utilities
def train_nerf(data_path, n_iters, batch_size, lr,
              n_samples, near, far, save_dir,
              multi_gpu=False, white_bkgd=True):
    os.makedirs(save_dir, exist_ok=True)
    device = torch.device('cuda' if torch.cuda.is_available()
                          else 'mps' if torch.backends.mps.is_available()
                          else 'cpu')
    data = np.load(data_path)
    images_train = data['images_train'] / 255.0
    c2ws_train = data['c2ws_train']
    images_val = data['images_val'] / 255.0
    c2ws_val = data['c2ws_val']
    K = data['K'] if 'K' in data else np.array([
        [data['focal'], 0, images_train.shape[2] / 2],
        [0, data['focal'], images_train.shape[1] / 2],
        [0, 0, 1]
    ])
    H, W = images_train.shape[1:3]

    model = NeRF(L_pos=10, L_dir=4, hidden_dim=256)
    if multi_gpu and torch.cuda.device_count() > 1:
        model = nn.DataParallel(model)
    model = model.to(device)

    optimizer = torch.optim.Adam(model.parameters(), lr=lr)
    criterion = nn.MSELoss()
    dataloader = NeRFDataloader(images_train, c2ws_train, K, H, W,
                                batch_size, device)

    psnr_history = []
    save_iters = [0, 100, 500, 1000, 2000, n_iters - 1]
    for it in tqdm(range(n_iters)):
        model.train()
        rays_o, rays_d, target_rgb = dataloader.sample()
        pred_rgb, _, _ = render_rays(model, rays_o, rays_d,
                                     n_samples, near, far,
                                     device=device, white_bkgd=white_bkgd,
                                     perturb=True)
        loss = criterion(pred_rgb, target_rgb)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        psnr = -10 * torch.log10(loss)
        psnr_history.append(psnr.item())

        if it in save_iters:
            eval_model = model.module if hasattr(model, 'module') else model
            eval_model.eval()
            with torch.no_grad():
                val_img = render_image(
                    eval_model,
                    torch.FloatTensor(c2ws_val[0]),
                    H, W, K,
                    n_samples=n_samples,
                    near=near, far=far,
                    device=device
                )
                plt.figure(figsize=(10, 5))
                plt.subplot(1, 2, 1)
                plt.imshow(images_val[0]); plt.title('Ground Truth'); plt.axis('off')
                plt.subplot(1, 2, 2)
                plt.imshow(np.clip(val_img, 0, 1))
                plt.title(f'Predicted (iter {it})'); plt.axis('off')
                plt.tight_layout()
                plt.savefig(f'{save_dir}/val_{it:04d}.png', dpi=150)
                plt.close()

    plt.figure(figsize=(10, 6))
    plt.plot(psnr_history); plt.xlabel('Iteration')
    plt.ylabel('PSNR (dB)'); plt.grid(True)
    plt.savefig(f'{save_dir}/psnr_curve.png', dpi=150); plt.close()

    model_to_save = model.module if hasattr(model, 'module') else model
    torch.save(model_to_save.state_dict(), f'{save_dir}/model.pth')
    return model, psnr_history

def render_validation_views(model, data_path, save_dir,
                            n_samples, near, far, device):
    os.makedirs(save_dir, exist_ok=True)
    data = np.load(data_path)
    images_val = data['images_val'] / 255.0
    c2ws_val = data['c2ws_val']
    K = data['K'] if 'K' in data else np.array([
        [data['focal'], 0, images_val.shape[2] / 2],
        [0, data['focal'], images_val.shape[1] / 2],
        [0, 0, 1]
    ])
    H, W = images_val.shape[1:3]

    eval_model = model.module if hasattr(model, 'module') else model
    eval_model.eval()
    with torch.no_grad():
        for i in range(len(images_val)):
            pred_img = render_image(eval_model,
                                    torch.FloatTensor(c2ws_val[i]),
                                    H, W, K,
                                    n_samples=n_samples,
                                    near=near, far=far,
                                    device=device)
            fig, axes = plt.subplots(1, 3, figsize=(15, 5))
            axes[0].imshow(images_val[i]); axes[0].set_title('GT'); axes[0].axis('off')
            axes[1].imshow(np.clip(pred_img, 0, 1))
            axes[1].set_title('Prediction'); axes[1].axis('off')
            axes[2].imshow(np.abs(images_val[i] - pred_img))
            axes[2].set_title('Error'); axes[2].axis('off')
            plt.tight_layout()
            plt.savefig(f'{save_dir}/val_{i:02d}.png', dpi=150, bbox_inches='tight')
            plt.close()

def run_lego(multi_gpu=False):
    save_dir = 'images/tmp/nerf'
    data_path = 'dataset/lego_200x200.npz'
    model, psnr_history = train_nerf(
        data_path=data_path,
        n_iters=5000,
        batch_size=10000,
        lr=5e-4,
        n_samples=64,
        near=2.0,
        far=6.0,
        save_dir=save_dir,
        multi_gpu=multi_gpu
    )
    device = torch.device('cuda' if torch.cuda.is_available()
                          else 'mps' if torch.backends.mps.is_available()
                          else 'cpu')
    render_validation_views(model, data_path, save_dir,
                            n_samples=64, near=2.0, far=6.0, device=device)
    render_novel_views(model, data_path, num_frames=120,
                       save_path=f'{save_dir}/novel_views.gif',
                       n_samples=64, near=2.0, far=6.0, device=device)
    print(f'Final PSNR: {psnr_history[-1]:.2f} dB')

Results

Using the Lego dataset (200×200), I trained with the following settings:

  • Iterations: 5,000
  • Batch size: 10,000 rays
  • Samples per ray: 64 (near=2.0, far=6.0)
  • Model: Same NeRF MLP from Part 2.4 with L_pos=10, L_dir=4
  • Optimizer: Adam @ 5e-4, white background

The final model reaches >23 dB PSNR. Below are the intermediate and final visualizations.

Lego training snapshots
▲ Predicted validation crops at iter 0 → 4999.
Ray visualization
▲ Rays + samples in viser (all cameras).
PSNR curve
▲ PSNR curve (validation view) during training.

Validation set renders:

Validation 00
Val 00
Validation 01
Val 01
Validation 02
Val 02
Validation 03
Val 03
Validation 04
Val 04
Validation 05
Val 05
Validation 06
Val 06
Validation 07
Val 07
Validation 08
Val 08
Validation 09
Val 09
Lego novel view GIF
▲ Novel-view animation (circular camera path) rendered from the trained NeRF.

Part 2.6: Training with Custom Data

I trained NeRF on my custom Miffy capture. I reused the same model architecture from Part 2.4 (identical NeRF MLP with L_pos=10, L_dir=4) and adjusted the training ranges to match the smaller capture volume.

  • Iterations: 8,000
  • Batch size: 8,192 rays
  • Samples per ray: 128 (near=0.05, far=0.65)
  • Optimizer: Adam @ 5e-4, white background
Miffy training predictions
▲ Predicted validation crops (iters 0 → 7999) for the Miffy dataset.
Miffy PSNR curve
▲ PSNR curve on the custom validation view.

Validation set renders:

Miffy Validation 00
Val 00
Miffy Validation 01
Val 01
Miffy Validation 02
Val 02
Miffy Validation 03
Val 03
Miffy novel views
▲ Novel-view animation rendered from the custom Miffy NeRF.

Debug Notes

I actually shot five different datasets before this run finally converged. The Lafufu set worked from day one, but my captures kept collapsing because the detector latched onto different ArUco IDs in the same sequence. In hindsight the issue was obvious: I taped down a sheet with six tags, so depending on perspective the “active” tag would change mid-frame.

Miffy debug collage
▲ Debug frames showing the tag ID jump when multiple tags are visible.

Once I re-shot with a single tag in the scene the pipeline behaved—proof that capture setup matters as much as the NeRF code. (Still not sure why the staff’s Lafufu dataset works with six tags.)

All datasets and trained weights for this whole project are open-sourced here: vaclisinc/2025_Computer_Vision/project_4.