Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Minor] Align layer norm eps value to PyTorch #221

Merged
merged 1 commit into from
Mar 1, 2022
Merged

[Minor] Align layer norm eps value to PyTorch #221

merged 1 commit into from
Mar 1, 2022

Conversation

blefaudeux
Copy link
Contributor

What does this PR do?

Follow up from #219, may not explain but better to stick to the same values all around

Before submitting

  • Did you have fun?
    • Make sure you had fun coding 🙃
  • Did you read the contributor guideline?
  • Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
    • N/A
  • Did you make sure to update the docs?
    • N/A
  • Did you write any new necessary tests?
    • N/A
  • Did you update the changelog? (if needed)
    • N/A

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 1, 2022
@blefaudeux blefaudeux requested review from fmassa and dianaml0 March 1, 2022 18:53
@blefaudeux blefaudeux linked an issue Mar 1, 2022 that may be closed by this pull request
xformers ViT-B ImageNet MAE + Deepnorm training instability #219
Open
@blefaudeux blefaudeux force-pushed the align_layer_norm_timm branch from edd670b to 6517b85 Compare March 1, 2022 19:25
@blefaudeux blefaudeux requested a review from jieru-hu March 1, 2022 22:47
@blefaudeux blefaudeux merged commit 2926d46 into align_mlp_fused_mlp Mar 1, 2022
blefaudeux added a commit that referenced this pull request Mar 2, 2022
@blefaudeux
[Minor] Expose bias options for both MLP and FusedMLP, use same defau…
d4c28fb 
…lts (#220)

* expose the bias options for both MLP and FusedMLP, use the same defaults
* using the same eps in layernorm as default torch (#221)
@blefaudeux blefaudeux mentioned this pull request Mar 8, 2022
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jieru-hu jieru-hu jieru-hu approved these changes

@fmassa fmassa Awaiting requested review from fmassa

@dianaml0 dianaml0 Awaiting requested review from dianaml0

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

xformers ViT-B ImageNet MAE + Deepnorm training instability
3 participants
@blefaudeux @jieru-hu @facebook-github-bot