$D^4NeRF$

PyTorch implementation of paper Detachable Novel Views Synthesis of Dynamic Scenes Using Distribution-Driven Neural Radiance Fields.

Novel View Synthesis

Abstract

Representing and synthesizing novel views in real-world dynamic scenes from casual monocular videos is a long-standing problem. Existing solutions typically approach dynamic scenes by applying geometry techniques or utilizing temporal information between several adjacent frames without considering the underlying background distribution in the entire scene or the transmittance over the ray dimension, limiting their performance on static and occlusion areas. Our approach Distribution-Driven neural radiance fields offers high-quality view synthesis and a 3D solution to Detach the background from the entire Dynamic scene, which is called $D^4$ NeRF. Specifically, it employs a neural representation to capture the scene distribution in the static background and a 6D-input NeRF to represent dynamic objects, respectively. Each ray sample is given an additional occlusion weight to indicate the transmittance lying in the static and dynamic components. We evaluate $D^4$ NeRF on public dynamic scenes and our urban driving scenes acquired from an autonomous-driving dataset. Extensive experiments demonstrate that our approach outperforms previous methods in rendering texture details and motion areas while also producing a clean static background.

Rendering Results

Comparison of novel view synthesis

Disentangled Results

Comparison on disentangled static background from entire scenes

The Entire Scene

The Decoupled Background

Getting Started

1. Setup&Dependency

The code is trained with Python == 3.8.8, Pytorch == 1.11.0 and CUDA == 11.3, the dependencies include:

• scikit-image
• opencv
• imageio
• cupy
• kornia
• configargparse

Then download NVIDIA Dynamic and Urban Driving datasets. The whole file structure should be:

D4NeRF
├── configs
├── logs
├── models
├── data
|  └── NVIDIA
|  └── URBAN
|  └── others
...

2. Train

python train.py --config configs/config_Handcart.txt 

3. Evaluation

The evaluation on NVIDIA dataset focuses on synthesis across different viewpoints, while evaluation on Urban driving dataset aims to interpolate time intervals (frames).

Evaluation on Urban Driving Scenes

python evaluation_NV.py --config configs/config_Balloon1.txt 

Evaluation on NVIDIA Dynamic Scenes

python evaluation_urban.py --config configs/config_Handcart.txt 

4. Novel view synthesis

fixed time and view interpolation:

python view_render.py --config configs/config_Handcart.txt --fixed_time --target_idx 15

time interpolation and fixed view:

python view_render.py --config configs/config_Handcart.txt --fixed_view --target_idx 15

time interpolation and view interpolation:

python view_render.py --config configs/config_Handcart.txt --no_fixed --target_idx 15

5. Create other datasets

Use COLMAP to acquire camera poses and intrinsics. Then download scripts to obtain the flow and depth estimation models, RAFT and Midas. The pre-trained weights have been added to the directory.

Pose transformation

python save_poses_nerf.py --data_path "/xxx/dense"  #data_path is the path of COLMAP estimation results.

Depth estimation

python run_midas.py --data_path "/xxx/dense" --resize_height 272

Flow estimation

python run_flows_video.py --model models/raft-things.pth --data_path /xxx/dense

Acknowledge

The code is built upon:

• NeRF_Pytorch
• NSFF
• DyNeRF

Thanks for their great work.