TL/UCP: Allow self copy in allgather using network loopback #1021
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
5e852a7 to
7e6c55d
Compare
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
d13b885 to
b3c2e80
Compare
yaeliyac
changed the title
|
ok to test |
nsarka
approved these changes
Dec 12, 2024
yaeliyac
force-pushed
the
origin/for_plugin
branch
3 times, most recently
from
482006f to
eecdebf
Compare
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
1d88c9b to
820794d
Compare
|
@Sergei-Lebedev why do we have to keep approving runs when force pushing?? Seems an outlier here |
yaeliyac
force-pushed
the
origin/for_plugin
branch
3 times, most recently
from
32b3ccd to
b1c875b
Compare
yaeliyac
changed the title
yaeliyac
force-pushed
the
origin/for_plugin
branch
3 times, most recently
from
2dc74f3 to
a2c5fee
Compare
| step = task->tagged.send_posted; | ||
| int iter = use_cuda ? tsize - 1 : tsize; //when using loopback tagged.send_posted has 1 more which will cause non-complete ring algorithm | ||
| while (task->tagged.send_posted < iter) { | ||
| step = use_cuda ? task->tagged.send_posted : task->tagged.send_posted - 1; //when using loopback tagged.send_posted has 1 more which will cause wrong calculation of send/recv |
There was a problem hiding this comment.
since loopback affects algorithm flow I would rather consider checking loopback copy result somehow differently, maybe add custom callback for ucp recv completion
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
a2c5fee to
5940100
Compare
samnordmann
reviewed
Dec 16, 2024
|
|
||
| if (UCC_INPROGRESS == ucc_tl_ucp_test(task)) { | ||
| return; | ||
| } | ||
| sendto = ucc_ep_map_eval(task->subset.map, (trank + 1) % tsize); | ||
| recvfrom = ucc_ep_map_eval(task->subset.map, (trank - 1 + tsize) % tsize); | ||
|
|
||
| while (task->tagged.send_posted < tsize - 1) { | ||
| step = task->tagged.send_posted; | ||
| int iter = use_loopback ? tsize : tsize - 1; //when using loopback tagged.send_posted has 1 more which will cause non-complete ring algorithm |
There was a problem hiding this comment.
declaration of variable goes to the beginning of the function
There was a problem hiding this comment.
I am not sure to understand the comment. What do you mean by "will cause non-complete ring algorithm"?
Can you use a more explicit name than iter for that variable?
|
@yaeliyac btw, for knomial inplace, we don't expect any change, right? Do we have an explanation for the degradation in the graph? |
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
5940100 to
6cb2e0e
Compare
yaeliyac
force-pushed
the
origin/for_plugin
branch
from
6cb2e0e to
5ba0039
Compare
ikryukov
reviewed
Dec 18, 2024
| if (ucc_unlikely(status != UCC_OK)) { | ||
| task->super.status = status; | ||
| return status; | ||
| if (!use_loopback) { |
There was a problem hiding this comment.
minor: suggestion for better readability is to use direct logic in cases:
if (use_loopback) {
...
} else {
...
}
| @@ -140,6 +140,10 @@ ucc_config_field_t ucc_tl_ucp_lib_config_table[] = { | |||
| ucc_offsetof(ucc_tl_ucp_lib_config_t, allgather_kn_radix), | |||
| UCC_CONFIG_TYPE_UINT}, | |||
|
|
|||
| {"ALLGATHER_USE_LOOPBACK", "0", "If set to 1 uses mc cuda copy, otherwise performs loopback for self copy", | |||
| return UCC_OK; | ||
| } | ||
|
|
||
|
|
| ucc_status_t status; | ||
|
|
||
|
|
||
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Add option for self copy loopback in out of place start for allgather algorithms: knomial, ring, neighbor, sparbit
Why ?
When using UCC plugin for NCCL, using cuda_memcpy might cause deadlock
How ?
Add UCC_TL_UCP_ALLGATHER_USE_LOOPBACK flag to control this
Tested on ISRAEL-1 with multiple test cases - showed good results and same performance with and without loopback