forward pass differences between "basic" img2img and img2img "alternative" script

[Ehplodor]

Hi,

I would like to better understand the difference between the following 3 flavors of img2img :

• "basic" img2img
• "alternatives" script

The latter (img2img alt) goes through N steps of progressive (latent) noise addition in order to recover back the noisy latent, thanks to some reversed sampler (currently, only reversed Euler is available).

About basic img2img, I wonder why it doesn't also goes through the N steps of iterative noise addition as in the "alternative" script ? If I understood well the information provided in this summary , and on wikipedia :

Gaussian noise is iteratively applied to the compressed latent representation during forward diffusion.[11]

So why does basic img2img is so fast compared to img2img alternative at computing iteratively noisy latent during the forward pass ?

[TKoestlerx]

At the beginning (noise=0) its the original image, at the other end of the scale its a pure gaussian(noise=1).

And the forward process can jump to any noise-level in 1step. There is no need to calculate noise level 0.01, 0.02.. just go for the 0.8 in one step.

img2imgalt needs to go back the ladder step by step. Because it has no easy gaussian to aim for. Its looking to find the original one..

[Ehplodor]

TY. Could you please elaborate on the fact that "there is no easy gaussian to aim for" in the alt script ?

Also, you mean that "basic" img2img forward pass is actually 1-step, hence the faster generation. But how can this be compatible with the fact that after that, one would typically do 20-50 steps of iterative denoising ? I ask this because for img2img "alt", it is typicall recommended to do as much forward steps as denoising steps.

[TKoestlerx]

From my understanding:

What img2imgalt does is, it tries to get the original noise the image was created with (given the 'old' prompt). And then de-noise it again with the 'new' prompt to get to an image that is close to the original.

img2img itself is noising itself into the direction of a random gaussian. And thats fast.

[Ehplodor]

Could you confirm to me that the "one-pass" img2img noise encoding happens at

stable-diffusion-webui/modules/processing.py

Line 830 in eeb0733

self.init_latent = self.sd_model.get_first_stage_encoding(self.sd_model.encode_first_stage(image))

?

[Ehplodor]

stable-diffusion-webui/modules/img2img.py

Line 89 in eeb0733

p = StableDiffusionProcessingImg2Img(

then

stable-diffusion-webui/modules/img2img.py

Line 139 in eeb0733

processed = process_images(p)

then

stable-diffusion-webui/modules/processing.py

Line 722 in eeb0733

class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):

then

stable-diffusion-webui/modules/processing.py

Line 830 in eeb0733

self.init_latent = self.sd_model.get_first_stage_encoding(self.sd_model.encode_first_stage(image))

[Ehplodor]

citation from https://lilianweng.github.io/posts/2021-07-11-diffusion-models/ : about the forward pass : (emphasis mine)

A nice property of the above process is that we can sample at any arbitrary time step in a closed form using reparameterization trick.

With that and the accompanying math proof, now I understand what you said earlier @TKoestlerx