Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Minor] FusedLayerNorm enabled by default in the factory #92

Merged
merged 1 commit into from
Nov 11, 2021
Merged

[Minor] FusedLayerNorm enabled by default in the factory #92

merged 1 commit into from
Nov 11, 2021

Conversation

blefaudeux
Copy link
Contributor

@blefaudeux blefaudeux commented Nov 10, 2021 •
edited
Loading

What does this PR do?

Enables FusedLayerNorm by default in the factory (if triton is available and the gpu is compatible), brings a little more speed and same accuracy in the microViT dummy test.

Somewhat related to #76

Before submitting

  • Did you have fun?
    • Make sure you had fun coding 🙃
  • Did you read the contributor guideline?
  • Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
    • N/A
  • Did you make sure to update the docs?
    • N/A
  • Did you write any new necessary tests?
    • N/A
  • Did you update the changelog? (if needed)
    • N/A

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 10, 2021
@blefaudeux blefaudeux force-pushed the triton_enable_layer_norm branch from 36b9e19 to bc0276c Compare November 11, 2021 04:43
@blefaudeux blefaudeux changed the base branch from triton_dropout_fused_bias to main November 11, 2021 04:44
@blefaudeux blefaudeux merged commit 1272b12 into main Nov 11, 2021
@blefaudeux blefaudeux deleted the triton_enable_layer_norm branch November 11, 2021 04:44
xwhan pushed a commit to xwhan/xformers that referenced this pull request Feb 8, 2022
@blefaudeux
[refactor] Cleaning up the masks, only expose attention masks in rele…
f849750 
…vant attentions (facebookresearch#92)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jieru-hu jieru-hu jieru-hu approved these changes

@fmassa fmassa Awaiting requested review from fmassa

@dianaml0 dianaml0 Awaiting requested review from dianaml0

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@blefaudeux @jieru-hu @facebook-github-bot