3D Human Reconstruction in the Wild with Synthetic Data Using Generative Models

🔰 Project Page, 📑 Paper, 🤗 HuggingFace Demo

Yongtao Ge^2,1, Wenjia Wang³, Yongfan Chen¹, Hao Chen¹, Chunhua Shen¹

¹Zhejiang University,   ²The University of Adelaide,   ³The University of Hong Kong  

📢 News

2024-03-22: Release pretrained weight on HuggingFace and demo code.

🛠️ Dependencies

pip install -r requirements.txt

🚀 Demo

1. download SDXL weight from https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0, put it in $SDXL_DIR

2. download fixed SDXL VAE weight from https://huggingface.co/madebyollin/sdxl-vae-fp16-fix, put it in $VAE_DIR

3. download the pretrained weight from huggingface repo https://huggingface.co/geyongtao/HumanWild and put it in $CONTROLNET_DIR

4. inference the demo code
python demo.py \
--normal_image_path 'data/controlnet_condition/00000_0_0002.png' \
--prompt 'a man lift his leg in the wild, photo realistic.' \
--controlnet_conditioning_scale 0.75 \
--output_dir './output_dir' \
--sdxl_dir $SDXL_DIR \
--vae_dir $VAE_DIR \
--normal_controlnet_dir $CONTROLNET_DIR \
--num_inference_steps 50 \
--num_images_per_prompt 4

🌟 Acknowledgement

We acknowledge the use of the following sources in our paper.

• BEDLAM
• SynBody
• LAION5B
• COYO-700M
• Hi4D

🎓 Citation

@article{ge2024humanwild,
    title={3D Human Reconstruction in the Wild with Synthetic Data Using Generative Models},
    author={Ge, Yongtao and Wang, Wenjia, and Chen, Yongfan, and Chen, Hao and Shen, Chunhua},
    journal={arXiv preprint arXiv:2403.11111},
    year={2024}
}