Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2-node AllReduce improvements #118

Merged
merged 64 commits into from
Jul 7, 2023
Merged

2-node AllReduce improvements #118

merged 64 commits into from
Jul 7, 2023

Conversation

chhwang
Copy link
Contributor

@chhwang chhwang commented Jul 5, 2023 •
edited
Loading

  • Added get() interfaces to SmChannel
  • Improved 2-node (8 gpus/node) AllReduce: algbw 139GB/s for 1GB (kernel 3) and 99GB/s for 48MB (kernel 4)
  • Fixed a FIFO perf bug
  • Several fixes & validations in mscclpp-test

Binyang2014 and others added 30 commits June 26, 2023 09:59
@Binyang2014
add reduce-scatter
a02f29c
@Binyang2014
fix
b64f6d2
@Binyang2014
debug
9c6eb7b
@Binyang2014
fix correctness issue
5126ddf
@Binyang2014
Fix
090e942
@Binyang2014
improve performance
b18cb83
@Binyang2014
multi init
7f6f180
@Binyang2014
fix
c44866b
@Binyang2014
naive reduce-scatter
ee85ae6
@Binyang2014
local allgather
21f5685
@Binyang2014
all reduce
e82a485
@Binyang2014
perf improve
f5c8670
@Binyang2014
more improve
54b7477
@Binyang2014
fix
9958ca0
@Binyang2014
add comments
504b24e
@Binyang2014
minor update
8ee39ef
@chhwang
Merge branch 'main' into binyli/allreduce-multinodes
daca40b
@chhwang
Merge with main
a844d10
@chhwang
Merge branch 'main' into binyli/allreduce-multinodes
b06043b
@chhwang
Remove unneeded code
484c6c9
@chhwang
Perf improved
a0934aa
@chhwang
Why does clang-format install fail?
d30439a
@Binyang2014
update pipeline
c7e9073
@Binyang2014
debug
02c439f
@Binyang2014
update perf num
2dec1dc
@chhwang
Lock GPU clock cycles
07700e9
@Binyang2014
update perf number
951cb6f
@Binyang2014
lock gpu frequency and update perf number
f14778a
@chhwang
Local AllGather using SmChannel
15b47f5
@chhwang
Merge branch 'binyli/allreduce-multinodes' of https://github.com/micr…
45999c4 
…osoft/mscclpp into binyli/allreduce-multinodes
@chhwang chhwang changed the title Binyli/allreduce multinodes 2-node AllReduce improvements Jul 5, 2023
@chhwang chhwang marked this pull request as ready for review July 5, 2023 05:49
@chhwang chhwang mentioned this pull request Jul 5, 2023
Closed
saeedmaleki
saeedmaleki approved these changes Jul 5, 2023
View reviewed changes
Copy link
Contributor

@saeedmaleki saeedmaleki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just minor changes.

include/mscclpp/fifo.hpp Show resolved Hide resolved
include/mscclpp/sm_channel.hpp Outdated Show resolved Hide resolved
include/mscclpp/sm_channel.hpp Outdated Show resolved Hide resolved
include/mscclpp/sm_channel.hpp Outdated Show resolved Hide resolved
include/mscclpp/sm_channel.hpp Outdated Show resolved Hide resolved
test/mscclpp-test/allreduce_test.cu Show resolved Hide resolved
test/mscclpp-test/allreduce_test.cu Outdated Show resolved Hide resolved
@Binyang2014 Binyang2014 force-pushed the binyli/allreduce-multinodes branch from 0ac9833 to 61449a8 Compare July 6, 2023 05:32
saeedmaleki
saeedmaleki reviewed Jul 6, 2023
View reviewed changes
Copy link
Contributor

@saeedmaleki saeedmaleki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a bit worried about the register usage of copy instructions. Did it matter for performance?

include/mscclpp/sm_channel.hpp Outdated Show resolved Hide resolved
include/mscclpp/sm_channel.hpp Show resolved Hide resolved
@chhwang chhwang enabled auto-merge (squash) July 7, 2023 05:47
@chhwang chhwang merged commit bb7b85a into main Jul 7, 2023
@chhwang chhwang deleted the binyli/allreduce-multinodes branch July 7, 2023 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@saeedmaleki saeedmaleki saeedmaleki approved these changes

@Binyang2014 Binyang2014 Binyang2014 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@chhwang @Binyang2014 @saeedmaleki