Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add PAG support for SD Img2Img #9463

Open
wants to merge 20 commits into
base: main
Choose a base branch
from

Conversation

SahilCarterr
Copy link

@SahilCarterr SahilCarterr commented Sep 18, 2024

What does this PR do?

Adds PAG (Perturbed-Attention Guidance) support for SD models (StableDiffusionPAGImg2ImgPipeline).Continuation of #7944

Fixes #8710 (partially)

Before submitting

Who can review?

@yiyixuxu
Anyone in the community is free to review the PR once the tests have passed.


Usage

from diffusers import AutoPipelineForImage2Image
from diffusers.utils import load_image, make_image_grid
import torch

# Set device to CUDA for GPU processing
device = "cuda"

# Load the image-to-image pipeline with pre-trained weights
pipe = AutoPipelineForImage2Image.from_pretrained("Lykon/dreamshaper-8", enable_pag=True)
pipe.to(device)

# Load an initial image
init_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png"
)

# Set a random generator with a specific seed
generator = torch.Generator("cuda").manual_seed(31)

# Define the prompt
prompt = "cat wizard"

# Generate the output image using the pipeline
image = pipe(prompt, image=init_image, generator=generator, pag_scale=0.4).images[0]

# Create a grid of the initial and generated images
make_image_grid([init_image, image], rows=1, cols=2)

cat_wizrd

Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! I left some comments
can we see some outputs too? (saw it)!

src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py Outdated Show resolved Hide resolved
src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py Outdated Show resolved Hide resolved
src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py Outdated Show resolved Hide resolved
tests/pipelines/pag/test_pag_sd_img2img.py Outdated Show resolved Hide resolved
tests/pipelines/pag/test_pag_sd_img2img.py Outdated Show resolved Hide resolved
@yiyixuxu yiyixuxu added the PAG label Sep 18, 2024
@yiyixuxu
Copy link
Collaborator

don't forget to call make style and make fix-copies too

@SahilCarterr
Copy link
Author

added "# copied from" , removed depreciated methods and fix test

Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! I think we still have one test needs to be removed, and then can merge!

), f"output is different from expected, {image_slice.flatten()}"

@slow
@require_torch_gpu
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test is not needed? (same as the one above it?)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should i remove StableDiffusionPAGImg2ImgPipelineIntegrationTests(unittest.TestCase): also ?

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yiyixuxu yiyixuxu mentioned this pull request Sep 20, 2024
6 tasks
@yiyixuxu
Copy link
Collaborator

@SahilCarterr can you look into failing tests?

@SahilCarterr
Copy link
Author

Screenshot 2024-09-25 at 10 47 58 PM

3 errors remaining to be fix

@SahilCarterr
Copy link
Author

Help Needed to fix the remaining errors

@SahilCarterr
Copy link
Author

The errors are now fixed @yiyixuxu
Screenshot 2024-10-06 at 1 25 56 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add PAG support to SD1.5
3 participants