Neural Radiance Fields

A neural radiance field ( NeRF ) is a simple fully connected network trained to reproduce input views of a single scene using a rendering loss. The network directly maps from spatial location and viewing direction (5D input) to colour and opacity (4D output), acting as the "volume" so we can use volume rendering to render new views differentiably.

Pipeline

Index

1. My Renders
2. Setup
3. Training
4. Evaluation
5. References and Citations

My Renders

Novel Views

3D Reconstruction

• Training graphs

MSE

PSNR

• Device used: TITAN X (Pascal) 250W / 12GiB RAM

• Time to train: 12h 30m

• Training Dashboard: https://wandb.ai/stanleyedward/LegoNeRF/runs/h6yb8pnb/overview

  Testing Metrics	Values
  Avg MSE	0.0012457877128773586
  Avg PSNR	29.200356294523996

Setup

1. Clone and cd into the repository:

   git clone https://github.com/stanleyedward/nerf_pytorch_lightning.git
   cd nerf_pytorch_lightning

2. Create and activate the conda environment:

   conda env create -f environment.yaml
   conda activate nerf

3. Add the dataset to the dataset/ directory:

• Download Lego Dataset: https://drive.google.com/drive/folders/1lrDkQanWtTznf48FCaW5lX9ToRdNDF1a

  dataset/
  ├── dataset_link.md
  └── lego
      ├── test
      ├── train
      ├── transforms_test.json
      ├── transforms_train.json
      ├── transforms_val.json
      └── val

Training

note: Setup should be complete

1. Change configurations

   In the config.py file

   """------------------------NeRF Config------------------------"""
   # data
   IMG_SIZE: int = 400
   BATCH_SIZE: int = 3072
   ...
   DEVICES: int = torch.cuda.device_count()
   MAX_EPOCHS: int = 17

2. Run the train.py script

   python train.py

Evaluation

note: Setup should be complete

1. Change configurations

   In the config.py file

   """------------------------NeRF Config------------------------"""
   ...
   #eval
   CKPT_DIR: str = "models/16_epoch_192_bins_400_nerf.ckpt" 
   CHUNK_SIZE: int = 20  # increase chunksize prevent CUDA out of memory errors
   OUTPUTS_DIR: str = "outputs" #folder you want to save the novel views in

2. Run the eval.py script

   python eval.py

References and Citations

• https://github.com/MaximeVandegar/Papers-in-100-Lines-of-Code/tree/main/NeRF_Representing_Scenes_as_Neural_Radiance_Fields_for_View_Synthesis

@misc{mildenhall2020nerf,
    title={NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis},
    author={Ben Mildenhall and Pratul P. Srinivasan and Matthew Tancik and Jonathan T. Barron and Ravi Ramamoorthi and Ren Ng},
    year={2020},
    eprint={2003.08934},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

@misc{queianchen_nerf,
  author={Quei-An, Chen},
  title={Nerf_pl: a pytorch-lightning implementation of NeRF},
  url={https://github.com/kwea123/nerf_pl/},
  year={2020},
}

@misc{lin2020nerfpytorch,
  title={NeRF-pytorch},
  author={Yen-Chen, Lin},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished={\url{https://github.com/yenchenlin/nerf-pytorch/}},
  year={2020}
}