Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SD3 Image-to-Image and Inpainting #7295

Merged
merged 15 commits into from
Nov 14, 2024
Merged

SD3 Image-to-Image and Inpainting #7295

merged 15 commits into from
Nov 14, 2024

Conversation

RyanJDick
Copy link
Collaborator

@RyanJDick RyanJDick commented Nov 7, 2024

Summary

Add support for SD3 image-to-image and inpainting. Similar to FLUX, the implementation supports fractional denoise_start/denoise_end for more fine-grained denoise strength control, and a gradient mask adjustment schedule for smoother inpainting seams.

Example

Workflow
image

Result
image

QA Instructions

  • Regression test of text-to-image
  • Test image-to-image without mask
  • Test that adjusting denoising_start allows fine-grained control of amount of change in image-to-image
  • Test inpainting with mask
  • Smoke test SD1, SDXL, FLUX image-to-image to make sure there was no regression with the frontend changes.

Merge Plan

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files labels Nov 7, 2024
@maryhipp maryhipp self-requested a review as a code owner November 8, 2024 18:43
@github-actions github-actions bot added the frontend PRs that change frontend files label Nov 8, 2024
@RyanJDick
Copy link
Collaborator Author

RyanJDick commented Nov 13, 2024

I ran through the tests listed in the PR description. Everything works, although inpainting/outpainting quality with SD3 is clearly worse than other model architectures (SD1, SDXL, FLUX). (The example in the PR description is cherry-picked.) I think it's ok to merge, but we may want to loop back and spend more time iterating on the inpainting implementation at some point.

@RyanJDick RyanJDick merged commit 0ba11e8 into main Nov 14, 2024
14 checks passed
@RyanJDick RyanJDick deleted the ryan/sd3-image-to-image branch November 14, 2024 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants