Low precision groupnorm #1976
Merged
Low precision groupnorm #1976
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mvpatel2000
requested review from
dblalock,
dskhudia and
nik-mosaic
as code owners
|
Thoughs on combining this with lowprecisonlayernorm and calling that lowPrecisionNorms? |
I would prefer to keep separate because we don't have guarantees of convergence with this, and there might be a case where one applies but another does not. While we have never observed issues with this, it is possible there are problems given AMP does not do this by default |
|
We can allow turning on/off each type based on an option. |
|
Is this scriptable? Can you export a model with LowPrecisionGroupnorm using torch.jit.script? |
dblalock
reviewed
Feb 17, 2023
composer/algorithms/low_precision_groupnorm/low_precision_groupnorm.py
Outdated
Show resolved
Hide resolved
…tel2000/composer into mvpatel2000/low-precision-groupnorm
|
Will switch to SimpleConvModel once #1991 merges |
dskhudia
approved these changes
Feb 24, 2023
nik-mosaic
suggested changes
Feb 24, 2023
composer/algorithms/low_precision_groupnorm/low_precision_groupnorm.py
Outdated
Show resolved
Hide resolved
composer/algorithms/low_precision_groupnorm/low_precision_groupnorm.py
Outdated
Show resolved
Hide resolved
nik-mosaic
approved these changes
Feb 24, 2023
|
I need a different logo... any suggestions for icon |
dblalock
approved these changes
Feb 25, 2023
composer/algorithms/low_precision_groupnorm/low_precision_groupnorm.py
Outdated
Show resolved
Hide resolved
composer/algorithms/low_precision_layernorm/low_precision_layernorm.py
Outdated
Show resolved
Hide resolved
composer/algorithms/low_precision_layernorm/low_precision_layernorm.py
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds low precision groupnorm. This is modeled after low precision layernorm and is useful for stable diffusion. We've seen about +5% throughput and memory savings with this with no significant impact to loss curves.
What issue(s) does this change relate to?
CO-1794