Efficient4D: Fast Dynamic 3D Object Generation from a Single-view Video

[Paper] | [Project]

Efficient4D: Fast Dynamic 3D Object Generation from a Single-view Video,
Zijie Pan, Zeyu Yang, Xiatian Zhu, Li Zhang
Arxiv preprint

Official implementation of "Efficient4D: Fast Dynamic 3D Object Generation from a Single-view Video".

Pipeline

Image generation results

demo.mp4

Install

conda create -n efficient4d python=3.9
conda activate efficient4d
pip install -r requirements.txt

Pretrained checkpoints

Following Syncdreamer to download checkpoints under ./ckpt.

Data

We provide one example case under ./data, please refer to Consistent4D for more data.

Tested environments

• Ubuntu 20 with torch 1.13 & CUDA 11.7 on eight A6000.

Usage

# set path
DEVICE="0,1,2,3"
N_FRAME=16  # 4 frames per GPU, 40G memory
INPUT="./data/walking_white_faced_egret"
OUTPUT="./output/walking_white_faced_egret"
OUTPUT_RECON="./recon_data/walking_white_faced_egret"

# generate pseudo muti-view and muti-frame consistent images
CUDA_VISIBLE_DEVICES=$DEVICE python generate.py \
  --input $INPUT \
  --output $OUTPUT \
  --crop_size 200 \
  --elevation 0 \
  --frame_num $N_FRAME \
  --smooth_filter \
  --decomposed_sampling \
  --is_cyc 0 \
  --seed 3407

# frame interpolation (optional)
for((i=0; i<$[N_FRAME-1]; i++))
do
  IMG1="$OUTPUT/$i.png"
  IMG2="$OUTPUT/$[i+1].png"
  SAVE_PATH="$OUTPUT/interp/${i}_$[i+1]"
  CUDA_VISIBLE_DEVICES=$DEVICE python frame_interpolation.py --img $IMG1 $IMG2 --cycle2 --save_path $SAVE_PATH
done

# preview
CUDA_VISIBLE_DEVICES=$DEVICE python concat_video.py --root_path $OUTPUT --num_frame $N_FRAME --interp

# confidence map
CUDA_VISIBLE_DEVICES=$DEVICE python video_confidence.py --root_path $OUTPUT

# pose
CUDA_VISIBLE_DEVICES=$DEVICE python data_convertor.py --input $OUTPUT --output $OUTPUT_RECON --confidence

We also provide a script run.sh to finish the above process in one command:

# bash run.sh DATA_NAME OUTPUT_NAME DEVICES N_FRAMES CYCLE
# We can set CYCLE=1 to enhance consistency if the input video is periodic 
# or the first frame and the last frame of the input video are similar.
bash run.sh walking_white_faced_egret walking_white_faced_egret 4,5,6,7 16 0

4D reconstruction

Now the preview videos are under output/DATA_NAME/preview and the data for reconstruction are under ./recon_data/DATA_NAME, please following 4DGS for reconstruction. Currently, the confidence map is not integrated into the 4DGS repo, we are trying to organize the simplified code in this repo. If the generated images have sufficient consistency with some seed, feel free to ignore it.

Acknowledgement

We have intensively borrow codes from the following repositories. Many thanks to the authors for sharing their codes.

• Syncdreamer
• RIFE
• Consistent4D
• 4DGS

BibTeX

If you find our repository useful, please consider giving it a star ⭐ and citing our paper in your work:

@article{pan2024efficient4d,
  title={Efficient4D: Fast Dynamic 3D Object Generation from a Single-view Video},
  author={Pan, Zijie and Yang, Zeyu and Zhu, Xiatian and Zhang, Li},
  journal={arXiv preprint arXiv 2401.08742},
  year={2024}
}