Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add IP Adapter Style & Composition Modes #6213

Merged
merged 22 commits into from
Apr 16, 2024
Merged

Add IP Adapter Style & Composition Modes #6213

merged 22 commits into from
Apr 16, 2024

Conversation

blessedcoolant
Copy link
Collaborator

@blessedcoolant blessedcoolant commented Apr 13, 2024 •
edited
Loading

Summary

Until now IP Adapter had complete control on the contents of the output. With this PR, users are now able to select "Style Only" or "Composition Only" to draw just the style or layout of the reference image.

Based off: https://arxiv.org/abs/2404.02733

New IP Method Option

  • Full - Both style and layout of the refence image are used.
  • Style Only - Only the style of the image is used
  • Composition Only - Only the composition of the image is used.

opera_0BkqZTwObO

Example Result

demo

Notes

  • Supports both SDXL and SD1.5

Testing

  • Just check and test if it works as expected with all IP Adapter models - both SDXL and SD1.5

Merge Plan

Good to merge once tested for all edge cases.

@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files frontend PRs that change frontend files labels Apr 13, 2024
@blessedcoolant blessedcoolant marked this pull request as ready for review April 13, 2024 06:37
@github-actions github-actions bot added the services PRs that change app services label Apr 13, 2024
@blessedcoolant blessedcoolant changed the title wip: Initial Implementation IP Adapter Style & Comp Modes Add IP Adapter Style & Composition Modes Apr 13, 2024
@blessedcoolant
Revert "chore: ruff fixes"
7a67fd6 
This reverts commit af36fe8.
@blessedcoolant
chore: ruff fixes
f2506cc 
Revert "chore: ruff fixes"

This reverts commit af36fe8.

Revert "chore: ruff fixes"

This reverts commit af36fe8.
@blessedcoolant
Merge branch 'ip-adapter-style-comp' of https://github.com/blessedcoo…
145a0b0 
…lant/InvokeAI into ip-adapter-style-comp
@blessedcoolant
chore: linter fixes
d4393e4
@blessedcoolant
refactor: fix a bunch of type issues in custom_attention
9cb0f63
@blessedcoolant
fix(experimental): Possible fix for conflict with regional embed leng…
8426f1e 
…th mismatch

Pushing this so people can test it out and see if this needs to be handled in a different way.
psychedelicious
psychedelicious previously requested changes Apr 14, 2024
View reviewed changes
Copy link
Collaborator

@psychedelicious psychedelicious left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple small comments, ML logic itself looks fine to me but probably best to get @RyanJDick to review that too.

invokeai/backend/stable_diffusion/diffusion/custom_atttention.py Outdated Show resolved Hide resolved
invokeai/backend/stable_diffusion/diffusion/custom_atttention.py Outdated Show resolved Hide resolved
invokeai/frontend/web/src/features/metadata/util/parsers.ts Outdated Show resolved Hide resolved
RyanJDick
RyanJDick requested changes Apr 15, 2024
View reviewed changes
Copy link
Collaborator

@RyanJDick RyanJDick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've found what I think are some significant bugs in the current implementation. I haven't taken a close look at custom_attention.py yet - I expect that fixing these bugs will have lots of trickle-down effects there.

invokeai/app/invocations/ip_adapter.py Outdated Show resolved Hide resolved
invokeai/backend/stable_diffusion/diffusion/custom_atttention.py Outdated Show resolved Hide resolved
invokeai/backend/stable_diffusion/diffusion/unet_attention_patcher.py Outdated Show resolved Hide resolved
RyanJDick
RyanJDick requested changes Apr 16, 2024
View reviewed changes
Copy link
Collaborator

@RyanJDick RyanJDick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few small things to fix, otherwise looks good to me.

invokeai/backend/stable_diffusion/diffusion/unet_attention_patcher.py Outdated Show resolved Hide resolved
invokeai/backend/stable_diffusion/diffusion/custom_atttention.py Show resolved Hide resolved
invokeai/backend/stable_diffusion/diffusion/custom_atttention.py Outdated Show resolved Hide resolved
invokeai/backend/stable_diffusion/diffusion/custom_atttention.py Outdated Show resolved Hide resolved
invokeai/backend/stable_diffusion/diffusion/custom_atttention.py Outdated Show resolved Hide resolved
invokeai/backend/stable_diffusion/diffusion/custom_atttention.py Show resolved Hide resolved
@RyanJDick
Copy link
Collaborator

The InstantStyle paper also suggested using embedding subtraction to isolate the style. Out of curiosity, did you try that at all?

@blessedcoolant
Copy link
Collaborator Author

The InstantStyle paper also suggested using embedding subtraction to isolate the style. Out of curiosity, did you try that at all?

No. Didn't check that. I'll take a look.

@blessedcoolant
Copy link
Collaborator Author

Pushed the changes you requested. Let me know if anything else needs changing.

@hipsterusername hipsterusername merged commit c828a4e into invoke-ai:main Apr 16, 2024
14 checks passed
@blessedcoolant blessedcoolant deleted the ip-adapter-style-comp branch May 4, 2024 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RyanJDick RyanJDick RyanJDick approved these changes

@hipsterusername hipsterusername hipsterusername approved these changes

@maryhipp maryhipp Awaiting requested review from maryhipp maryhipp is a code owner

@lstein lstein Awaiting requested review from lstein lstein is a code owner

@brandonrising brandonrising Awaiting requested review from brandonrising brandonrising is a code owner

@psychedelicious psychedelicious Awaiting requested review from psychedelicious psychedelicious is a code owner

Assignees
No one assigned
Labels
backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files services PRs that change app services
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@blessedcoolant @RyanJDick @psychedelicious @hipsterusername