Follow-Your-Emoji
Fine-Controllable and Expressive Freestyle Portrait Animation

Yue Ma*, Hongyu Liu*, Hongfa Wang*, Heng Pan*, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu and Qifeng Chen

is Accpeted by Siggraph Asia 2024

📣 Updates

• [2024.07.31] 🔥 Release OpenXLab, thanks for keyhsw development!
• [2024.07.21] 🔥 Release Colab, thanks for daswer123!
• [2024.07.18] 🔥 Release inference code, config and checkpoints!
• [2024.06.07] 🔥 Release Paper and Project page!

🤪 Gallery

We present Follow-Your-Emoji, a diffusion-based framework for portrait animation, which animates a reference portrait with target landmark sequences.

🤪 Getting Started

1. Clone the code and prepare the environment

pip install -r requirements.txt

2. Download pretrained weights

[FollowYourEmoji] We also provide our pretrained checkpoints in Huggingface. you could download them and put them into checkpoints folder to inference our model.

• base model: lambdalabs/sd-image-variations-diffusers
• vae: sd-vae-ft-mse
• AnimateDiff: AnimateDiff

Finally, these pretrained models should be organized as follows:

pretrained_models
├── AnimateDiff
│   └── mm_sd_v15_v2.ckpt
├── follow-your-emoji
│   ├── lmk_guider.pth
│   ├── referencenet.pth
│   └── unet.pth
├── sd-image-variations-diffusers
│   ├── alias-montage.jpg
│   ├── default-montage.jpg
│   ├── earring.jpg
│   ├── feature_extractor
│   │   └── preprocessor_config.json
│   ├── image_encoder
│   │   ├── config.json
│   │   └── pytorch_model.bin
│   ├── inputs.jpg
│   ├── model_index.json
│   ├── README.md
│   ├── safety_checker
│   │   ├── config.json
│   │   └── pytorch_model.bin
│   ├── scheduler
│   │   └── scheduler_config.json
│   ├── unet
│   │   ├── config.json
│   │   └── diffusion_pytorch_model.bin
│   ├── v1-montage.jpg
│   ├── v2-montage.jpg
│   └── vae
│       ├── config.json
│       └── diffusion_pytorch_model.bin
└── sd-vae-ft-mse
    ├── config.json
    ├── diffusion_pytorch_model.bin
    ├── diffusion_pytorch_model.safetensors
    └── README.md

3. Inference 🚀

bash infer.sh

CUDA_VISIBLE_DEVICES=0 python3 -m torch.distributed.run \
    --nnodes 1 \
    --master_addr $LOCAL_IP \
    --master_port 12345 \
    --node_rank 0 \
    --nproc_per_node 1 \
    infer.py \
    --config ./configs/infer.yaml \
    --model_path /path/to/model \
    --input_path your_own_images_path \
    --lmk_path ./inference_temple/test_temple.npy  \
    --output_path your_own_output_path

🤪 Make Your Emoji

You can make your own emoji using our model. First, you need to make your emoji temple using MediaPipe. We provide the script in make_temple.ipynb. You can replace the video path with your own emoji video and generate the .npy file.

CUDA_VISIBLE_DEVICES=0 python3 -m torch.distributed.run \
    --nnodes 1 \
    --master_addr $LOCAL_IP \
    --master_port 12345 \
    --node_rank 0 \
    --nproc_per_node 1 \
    infer.py \
    --config ./configs/infer.yaml \
    --model_path /path/to/model \
    --input_path your_own_images_path \
    --lmk_path  your_own_temple_path \
    --output_path your_own_output_path

👨‍👩‍👧‍👦 Follow Family

Follow-Your-Pose: Pose-Guided text-to-Video Generation.

Follow-Your-Click: Open-domain Regional image animation via Short Prompts.

Follow-Your-Handle: Controllable Video Editing via Control Handle Transformations.

Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation.

Citation 💖

If you find Follow-Your-Emoji useful for your research, welcome to 🌟 this repo and cite our work using the following BibTeX:

@article{ma2024follow,
  title={Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation},
  author={Ma, Yue and Liu, Hongyu and Wang, Hongfa and Pan, Heng and He, Yingqing and Yuan, Junkun and Zeng, Ailing and Cai, Chengfei and Shum, Heung-Yeung and Liu, Wei and others},
  journal={arXiv preprint arXiv:2406.01900},
  year={2024}
}