Add imagic to community pipelines

[MarkRich]

Part of #895 and bigger story #841

Followed the rough code / parameters given here: https://github.com/justinpinkney/stable-diffusion/blob/main/notebooks/imagic.ipynb.

A few notes for reviews:

• Seems like a bad idea to modify the stable diffusion pipeline to take text embeddings; that said it wasn't clear how we plan to re-use this pipeline given these custom generated embeddings. Any advice on how to pass these in? Or is modification given reasonable?
• Also seems like a bad idea to define the learning rates where I do. That said this is a bit of an odd stable diffusion pipeline since we're training in the __call__, so not sure where these are expected to go.
• Similar comment as in Add Composable diffusion to community pipeline examples  #951: Test condition given in readme doesn't work locally and need to make a few changes to make it work locally, but I expect this is only an issue w/ local testing and/or I am missing something w.r.t. how these scripts are intended to be tested.

Results:

Requires 24gb of vram and takes about 7-10 minutes on a 3090, though apparently it's 30g vram in 5pm on an a100 in original script. So reasonable performance?

Initial Image:

Prompt: "A photo of Barack Obama smiling with a big grin"

Image from just text embedding:

Image after text embeddings have been optimized:

Final image at alpha = 0.8

Final Image at alpha = 1.5

Final Image at alpha = 2.

Looking forward to any comments!

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Great job @MarkRich!

The design is generally fine with me! Also related to #955 - seems like there are multiple use cases for custom text_embeddings already

@patil-suraj could you do a more in-depth review?

[patil-suraj]

Very cool @MarkRich , thanks a lot for adding the feature

Th pr looking really good! Just left a few nits.

And I'm not sure yet, if we should modify the StableDiffusionPipeline to allow text_embeddinsg, we are discussing it here #955

For now, since we are adding a custom pipeline, I would suggest we could add to functions to the pipeline.

1. pipeline.train to train the embeddings and mode
2. pipeline.__call__ or pipeline.generate to generate the images.

wdyt @patrickvonplaten

[patil-suraj]

Maybe also allow the option to use 8 but optimizer

[MarkRich]

Addressed all your comments @patil-suraj aside from 8-bit optimization which may take slightly longer for to instrument due to an unrelated error. Let me know if you have any other comments!

[patil-suraj]

Thanks a lot for addressing the comments @MarkRich !Looks good, will give it try now and then merge soon :)

[patil-suraj]

The example works great, I just two more comments, then it should be good to merge :)

[patrickvonplaten]

Looks good! Thanks a lot :-)
@patil-suraj feel free to merge whenever

[MarkRich]

Updated with comments from the review; should be good to go!

[patil-suraj]

Thanks a lot @MarkRich ! The tests failures are unrelated, merging!

[zhongyi-zhou]

@MarkRich Thanks for the amazing code!
I have one question on this line:

diffusers/examples/community/imagic_stable_diffusion.py

Line 312 in f9cfb5a

noise_pred = self.unet(noisy_latents, timesteps, text_embeddings).sample

Why it is text_embeddings instead of text_embeddings_orig?

According to the Imagic paper, in the "model fine-tuning" stage, it says

• "We fine-tune them with the same reconstruction loss, but conditioned on $e_{tgt}$, as $e_{opt}$ is optimized for the base model only" (Page 5; the second column)
  

From my understanding $e_{tgt}$ is text_embeddings_orig and $e_{opt}$ is text_embeddings in this code.
In practice, these two values should be very similar, and probably that's the reason why the code still work amazingly well.

Please correct me if I am wrong. Thanks!

[shiranzada]

Thanks @zhongyi-zhou, we condition on e_tgt during finetuning only for the super resolution models of Imagen. This part is not relevant for Stable Diffusion.

For the base model (Imagen-64 or LDM) we condition on e_opt during finetuning rather than on e_tgt in order to overfit the image (for e_opt)