Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update blocksparse to use additive attention, more consistent with core #85

Merged
merged 1 commit into from
Nov 8, 2021

Conversation

dianaml0
Copy link
Contributor

@dianaml0 dianaml0 commented Nov 7, 2021

What does this PR do?

Update blocksparse to use additive attention, this would be more consistent with the core attention masking, which is now only additive or boolean. Perhaps we can allow either options to be specified?

Before submitting

  • Did you have fun?
    • Make sure you had fun coding 🙃
  • Did you read the contributor guideline?
  • Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
    • N/A
  • Did you make sure to update the docs?
    • N/A
  • Did you write any new necessary tests?
    • N/A
  • Did you update the changelog? (if needed)
    • N/A

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 7, 2021
@codecov-commenter
Copy link

Codecov Report

Merging #85 (8f3c01e) into main (ec26827) will not change coverage.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main      #85   +/-   ##
=======================================
  Coverage   87.08%   87.08%           
=======================================
  Files          50       50           
  Lines        2447     2447           
=======================================
  Hits         2131     2131           
  Misses        316      316           
Flag Coverage Δ
Python 87.08% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
xformers/components/attention/blocksparse.py 90.27% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ec26827...8f3c01e. Read the comment docs.

Copy link
Contributor

@blefaudeux blefaudeux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thank you @dianaml0 !

@dianaml0 dianaml0 merged commit a56c542 into main Nov 8, 2021
@dianaml0 dianaml0 deleted the blocksparse_mask_update branch November 23, 2021 21:01
xwhan pushed a commit to xwhan/xformers that referenced this pull request Feb 8, 2022
* add causal masking option
* minor, caching the causal masks and moving them to device

Co-authored-by: Benjamin Lefaudeux <benjamin.lefaudeux@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants