TL/NCCL: make team init non blocking #772
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shimmybalsam
force-pushed
the
nccl_team_create_nb
branch
2 times, most recently
from
079f631 to
fc6ca00
Compare
|
Note that non-blocking flag affects all NCCL apis, including communications and comm destroy. In particular https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/api/comms.html#ncclcommfinalize has to be used in communicator release/destroy. |
shimmybalsam
force-pushed
the
nccl_team_create_nb
branch
from
d41920e to
283fd56
Compare
samnordmann
approved these changes
May 17, 2023
shimmybalsam
force-pushed
the
nccl_team_create_nb
branch
from
283fd56 to
6e98b6e
Compare
shimmybalsam
force-pushed
the
nccl_team_create_nb
branch
2 times, most recently
from
8542cdb to
6558226
Compare
|
Just to clarify - this will make UCC configured and built with NCCL > 2.14.3 incompatible with NCCL < 2.14.3 at the runtime, right? Not complaining, just something worth mentioning in the docs. |
@marsaev Yes since nccl supports ncclCommInitRankConfig only from 2.14.3. I will make sure to add to docs. |
Sergei-Lebedev
approved these changes
Jun 1, 2023
shimmybalsam
force-pushed
the
nccl_team_create_nb
branch
from
6558226 to
e957b14
Compare
bureddy
approved these changes
Jun 7, 2023
shimmybalsam
force-pushed
the
nccl_team_create_nb
branch
from
e957b14 to
c558fe3
Compare
janjust
pushed a commit
to janjust/ucc
that referenced
this pull request
Jan 31, 2024
* TL/NCCL: make team init non blocking * TL/NCCL: support by version and nb finalize * REVIEW: code review fixes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Make nccl team create non blocking
Why ?
All UCC TL's team create are supposed to be non blocking, but TL/NCCL was using old NCCL API which was blocking.
How ?
Updated blocking ncclCommInitRank() to non blocking ncclCommInitRankConfig()