Deep t2i

My Experients of text to image using deep learning

1. Generate 256x256 bird image from natural text.
2. Generate 64x64 anime faces image from specific tags.

Video

I made a explanation video using chinese:

Training and Result

Please check these notebooks which contain the full training process. (20a_Train_Anime_Heads_DAMSM.ipynb, 20b_Train_Anime_Heads_GAN.ipynb, 20c_Train_Birds_DAMSM.ipynb, 20d_Train_Birds_GAN.ipynb )

Notice that for birds dataset, the results are not always good. The followings are which I choosed that looks good.

You can check birds_results or anime_heads_results to get the result images during training process for each epoch.

Data

• Birds:
  • CUB-200
  • Birds Captions from AttnGAN
• Anime faces
  • Simple anime faces data
  • This data is from this course

Model Architecture

DAMSM

GAN

The architecture design is largely inspired by Attn-GAN, MSG-GAN, SA-GAN and Pro-GAN.

1. Skip connection from MSG-GAN
2. Attention and DAMSM from AttnGAN,
3. SelfAtten, Spectral norm, Batch Norm from SAGAN
4. Architecture and running average of the generator weights from ProGAN but add some residual and attention

Please check this notebook to get more details of my model architecture. ( 03a_model.ipynb )

Check these notebooks to get details of the loss and training. ( 04a_trainer_DAMSM.ipynb, 04b_trainer_GAN.ipynb )

Pretrained Model

Exported Anime Face Model
Exported Birds Model

See 21a_example_inference.ipynb for inference example

Flask Server and Simple Demo Site

• Flask Server that deployed on Google Cloud Run
• Simple Demo Site: Simple web interface for the flask server

How to Develop

• Clone this repository
• Run make install to install deep_t2i
• Data:
  • Download CUB-200-2011 and put it to ./data/full_data/CUB-200-2011.tgz
  • Download Birds Captions from AttnGAN and put it to ./data/full_data/birds.zip
  • Download Simple anime faces data and put it to ./data/full_data/data.zip
  • Run python3 ./scripts/birds_data.py and python3 ./scripts/anime_heads_data.py
• Make some change in nbs/xxx.ipynb. You can add documentation in notebooks!!!
• Run make build to generate code from jupyter notebooks to ./deep_t2i
• Run make test to do some unit test
• Run make build-all to also generate documentation from notebooks
• Git push

How to Train

• Clone this repository
• Run make install to install deep_t2i
• Data:
  • Download CUB-200-2011 and put it to ./data/full_data/CUB-200-2011.tgz
  • Download Birds Captions from AttnGAN and put it to ./data/full_data/birds.zip
  • Download Simple anime faces data and put it to ./data/full_data/data.zip
  • Run python3 ./scripts/birds_data.py and python3 ./scripts/anime_heads_data.py
• See ./nbs/20c_Train_Birds_DAMSM.ipynb for training DAMSM
• See ./nbs/20d_Train_Birds_GAN.ipynb for training and getting exported model
• See ./nbs/21a_example_inference.ipynb for inference