MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

Zhongcong Xu · Jianfeng Zhang · Jun Hao Liew · Hanshu Yan · Jia-Wei Liu · Chenxu Zhang · Jiashi Feng · Mike Zheng Shou


National University of Singapore   |   ByteDance

📢 News

• [2023.12.8] Add loading local safetensors or ckpt,you can change config/prompts/animation.yaml about pretrained_model_path for your local SD1.5 model. such as "D:\\stablediffusion-webui\\models\Stable-diffusion\\v1-5-pruned.ckpt"
• [2023.12.4] Release inference code and gradio demo. We are working to improve MagicAnimate, stay tuned!
• [2023.11.23] Release MagicAnimate paper and project page.

⚒️ Installation

prerequisites: python>=3.8, CUDA>=11.3, ffmpeg and git.

Python and Git:

• Python 3.10.11: https://www.python.org/ftp/python/3.10.11/python-3.10.11-amd64.exe

• git: https://git-scm.com/download/win

• Install ffmpeg for your operating system (https://www.geeksforgeeks.org/how-to-install-ffmpeg-on-windows/)

  notice:step 4 use windows system Set Enviroment Path.

Give unrestricted script access to powershell so venv can work:

• Open an administrator powershell window
• Type Set-ExecutionPolicy Unrestricted and answer A
• Close admin powershell window

git clone --recurse-submodules https://github.com/sdbds/magic-animate-for-windows/

Install with Powershell run install.ps1 or install-cn.ps1(for Chinese)

Use local model

Add loading local safetensors or ckpt,you can change config/prompts/animation.yaml about pretrained_model_path for your local SD1.5 model. such as "D:\\stablediffusion-webui\\models\Stable-diffusion\\v1-5-pruned.ckpt"

🎨 Gradio Demo

Online Gradio Demo:

Try our online gradio demo quickly.

Local Gradio Demo:

Launch local gradio demo on single GPU:

Powershell run with run_gui.ps1

Launch local gradio demo if you have multiple GPUs:

Edit run_gui.ps1 set $mutil_gpu=1 then run.

Then open gradio demo in local browser.

🙏 Acknowledgements

We would like to thank AK(@_akhaliq) and huggingface team for the help of setting up oneline gradio demo.

🎓 Citation

If you find this codebase useful for your research, please use the following entry.

@inproceedings{xu2023magicanimate,
    author    = {Xu, Zhongcong and Zhang, Jianfeng and Liew, Jun Hao and Yan, Hanshu and Liu, Jia-Wei and Zhang, Chenxu and Feng, Jiashi and Shou, Mike Zheng},
    title     = {MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model},
    booktitle = {arXiv},
    year      = {2023}
}