This repository is an implementation of "Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels", in NeurIPS 2024.
Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels
Yikai Wang*, Xinzhou Wang*, Zilong Chen, Zhengyi Wang, Fuchun Sun, Jun Zhu
See accompanying videos in our project page for better visual quality.
First, specify the experiment name,
seqname=cheetah
Then, create a seqname folder under database/raw and put a video (e.g., 0.mp4) in it.
Stage 1, data preprocess, including data cropping, optical flow, DINO feature, rough poses, etc,
python scripts/run_preprocess.py $seqname $seqname other "0"
Stage 2, optimize neural sdf,
python lab4d/train.py --seqname $seqname --logname base --fg_motion bob --num_rounds 21 --rgb_timefree --rgb_dirfree
Stage 3, optimize Gaussian surfels,
python lab4d/train.py --seqname ${seqname} --logname gs-frzwarp --fg_motion gs-bob --num_rounds 61 --load_path logdir/${seqname}-base/ckpt_0020.pth --gs_init_mesh logdir/${seqname}-base/021-fg-geo.obj --imgs_per_gpu 1 --pixels_per_image -1 --eval_res 256 --rgb_timefree --rgb_dirfree --rgb_loss_only --gs_optim_warp=False --data_prefix full --force_center_cam
Render after training,
python lab4d/render.py --flagfile=logdir/${seqname}-gs-frzwarp/opts.log --load_suffix latest --render_res 512
- DGS Refinement
- Benchmarks
Our code is built based on 2DGS and Lab4D. We thank the authors for their great repos.
@inproceedings{wang2024vidu4d,
title={Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels},
author={Yikai Wang and Xinzhou Wang and Zilong Chen and Zhengyi Wang and Fuchun Sun and Jun Zhu},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2024}
}