Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[deps] update compel to fix black output with weight=0; also use new downweighting algorithm #2961

Conversation

damian0815
Copy link
Contributor

@damian0815 damian0815 commented Mar 15, 2023

Update compel to 1.0.0.

This fixes #2832.

It also changes the way downweighting is applied. In particular, downweighting should now be much better and more controllable.

From the compel changelog:

Downweighting now works by applying an attention mask to remove the downweighted tokens, rather than literally removing them from the sequence. This behaviour is the default, but the old behaviour can be re-enabled by passing downweight_mode=DownweightMode.REMOVE on init of the Compel instance.

Formerly, downweighting a token worked by both multiplying the weighting of the token's embedding, and doing an inverse-weighted blend with a copy of the token sequence that had the downweighted tokens removed. The intuition is that as weight approaches zero, the tokens being downweighted should be actually removed from the sequence. However, removing the tokens resulted in the positioning of all downstream tokens becoming messed up. The blend ended up blending a lot more than just the tokens in question.

As of v1.0.0, taking advice from @keturn and @bonlime (damian0815/compel#7) the procedure is by default different. Downweighting still involves a blend but what is blended is a version of the token sequence with the downweighted tokens masked out, rather than removed. This correctly preserves positioning embeddings of the other tokens.

@blessedcoolant blessedcoolant merged commit c341dca into invoke-ai:main Mar 16, 2023
@damian0815 damian0815 changed the title update compel to fix black screens and use new downweighting algorithm [deps] update compel to fix black output with weight=0; also use new downweighting algorithm Mar 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[bug]: Prompt with term weighted 0 results in an all-black image
2 participants