New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TL/MLX5: revise team and ctx init #815

Merged

Sergei-Lebedev merged 14 commits into openucx:master from samnordmann:tl_mlx5/fix_team_creation

Aug 22, 2023

Collaborator

samnordmann commented Jul 31, 2023 •

edited

Loading

What

Revise TL/MLX5 context and team init.

Allow other collective to run if a2a setup fails
Handling device memory allocation error, and other errors happening during team alltoall init
allocate device memory atomics before device memory pool
change setup/cleanup error log to debug log

Along the way, we fix two important bugs

fix pd deallocation https://redmine.mellanox.com/issues/3558239
fix socket closing https://redmine.mellanox.com/issues/3564403

samnordmann force-pushed the tl_mlx5/fix_team_creation branch 2 times, most recently from 1575241 to 2b0dfb3 Compare

July 31, 2023 10:06

samnordmann requested review from shimmybalsam, Sergei-Lebedev and MamziB

July 31, 2023 11:20

samnordmann added the Ready-for-Review label

MamziB reviewed

View reviewed changes

src/components/tl/mlx5/tl_mlx5_context.c Outdated Show resolved Hide resolved

MamziB reviewed

View reviewed changes

src/components/tl/mlx5/tl_mlx5_context.c Show resolved Hide resolved

MamziB reviewed

View reviewed changes

src/components/tl/mlx5/tl_mlx5_context.c Outdated Show resolved Hide resolved

MamziB reviewed

View reviewed changes

src/components/tl/mlx5/tl_mlx5_context.c Outdated Show resolved Hide resolved

MamziB reviewed

View reviewed changes

src/components/tl/mlx5/tl_mlx5_context.c Outdated Show resolved Hide resolved

MamziB reviewed

View reviewed changes

src/components/tl/mlx5/tl_mlx5_team.c Outdated Show resolved Hide resolved

MamziB reviewed

View reviewed changes

src/components/tl/mlx5/tl_mlx5_team.c Outdated Show resolved Hide resolved

MamziB reviewed

View reviewed changes

src/components/tl/mlx5/tl_mlx5.h Outdated Show resolved Hide resolved

samnordmann force-pushed the tl_mlx5/fix_team_creation branch from 26c7d95 to 9008049 Compare

August 7, 2023 12:34

samnordmann requested a review from MamziB

August 7, 2023 12:35

samnordmann force-pushed the tl_mlx5/fix_team_creation branch from 9008049 to 9744aec Compare

August 7, 2023 12:45

shimmybalsam approved these changes

View reviewed changes

src/components/tl/mlx5/alltoall/alltoall.c Outdated Show resolved Hide resolved

src/components/tl/mlx5/tl_mlx5_context.c Outdated Show resolved Hide resolved

src/components/tl/mlx5/tl_mlx5_team.c Show resolved Hide resolved

shimmybalsam reviewed

View reviewed changes

src/components/tl/mlx5/tl_mlx5_context.c Show resolved Hide resolved

shimmybalsam approved these changes

View reviewed changes

MamziB approved these changes

View reviewed changes

samnordmann requested a review from bureddy

August 10, 2023 11:24

Sergei-Lebedev reviewed

View reviewed changes

src/components/tl/mlx5/alltoall/alltoall.c Outdated Show resolved Hide resolved

src/components/tl/mlx5/tl_mlx5_team.c Outdated Show resolved Hide resolved

src/components/tl/mlx5/tl_mlx5_team.c Outdated Show resolved Hide resolved

src/components/tl/mlx5/tl_mlx5_team.c Outdated Show resolved Hide resolved

samnordmann added 7 commits

August 17, 2023 20:14


          TL/MLX5: revise team and ctx init

24f8443


          TL/MLX5: change setup/cleanup error log to debug

196412a


          TL/MLX5: minor reviews

6c48bf2


          CODESTYLE: clang-tidy

55940b0


          TL/MLX5: alloc dm atomics before mpool

5937c62


          CODESTYLE: clang-tidy

b6507a1


          TL/MLX5: fix bug with with socket

samnordmann added 4 commits

August 17, 2023 20:14


          REVIEW: minor comments

c625b1d


          TL/MLX5: disable coverity issue

b4afb68


          REVIEW: minor comments

2ff3c45


          TL/MLX5: fix socket closing

855f390

samnordmann force-pushed the tl_mlx5/fix_team_creation branch from 2d8cc32 to 855f390 Compare

August 17, 2023 17:14

samnordmann requested a review from Sergei-Lebedev

August 17, 2023 17:15

Sergei-Lebedev reviewed

View reviewed changes

src/components/tl/mlx5/tl_mlx5_team.c Show resolved Hide resolved

src/components/tl/mlx5/tl_mlx5_team.c Show resolved Hide resolved

samnordmann added 3 commits

August 21, 2023 13:17


          TL/MLX5: score map if a2a not avail

af88743


          Merge branch 'master' of https://github.com/openucx/ucc into tl_mlx5/…

b8ea9dc

…fix_team_creation


          CODESTYLE: clang format

bff0c7e

samnordmann force-pushed the tl_mlx5/fix_team_creation branch from ae74c70 to bff0c7e Compare

August 21, 2023 10:26

samnordmann requested a review from Sergei-Lebedev

August 21, 2023 10:41

Sergei-Lebedev approved these changes

View reviewed changes

samnordmann requested a review from manjugv

August 21, 2023 15:21

bureddy approved these changes

View reviewed changes

manjugv approved these changes

View reviewed changes

Sergei-Lebedev merged commit cd7175c into openucx:master

nsarka pushed a commit to nsarka/ucc that referenced this pull request


          TL/MLX5: revise team and ctx init (openucx#815)

38450c7

* TL/MLX5: revise team and ctx init

* TL/MLX5: change setup/cleanup error log to debug

* TL/MLX5: minor reviews

* CODESTYLE: clang-tidy

* TL/MLX5: alloc dm atomics before mpool

* CODESTYLE: clang-tidy

* TL/MLX5: fix bug with with socket

* REVIEW: minor comments

* TL/MLX5: disable coverity issue

* REVIEW: minor comments

* TL/MLX5: fix socket closing

* TL/MLX5: score map if a2a not avail

* CODESTYLE: clang format

nsarka pushed a commit to nsarka/ucc that referenced this pull request


          TL/MLX5: revise team and ctx init (openucx#815)

a5c7161

* TL/MLX5: revise team and ctx init

* TL/MLX5: change setup/cleanup error log to debug

* TL/MLX5: minor reviews

* CODESTYLE: clang-tidy

* TL/MLX5: alloc dm atomics before mpool

* CODESTYLE: clang-tidy

* TL/MLX5: fix bug with with socket

* REVIEW: minor comments

* TL/MLX5: disable coverity issue

* REVIEW: minor comments

* TL/MLX5: fix socket closing

* TL/MLX5: score map if a2a not avail

* CODESTYLE: clang format

janjust pushed a commit to janjust/ucc that referenced this pull request


          TL/MLX5: revise team and ctx init (openucx#815)

5653ab9

* TL/MLX5: revise team and ctx init

* TL/MLX5: change setup/cleanup error log to debug

* TL/MLX5: minor reviews

* CODESTYLE: clang-tidy

* TL/MLX5: alloc dm atomics before mpool

* CODESTYLE: clang-tidy

* TL/MLX5: fix bug with with socket

* REVIEW: minor comments

* TL/MLX5: disable coverity issue

* REVIEW: minor comments

* TL/MLX5: fix socket closing

* TL/MLX5: score map if a2a not avail

* CODESTYLE: clang format

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

MamziB MamziB approved these changes

shimmybalsam shimmybalsam approved these changes

Sergei-Lebedev Sergei-Lebedev approved these changes

bureddy bureddy approved these changes

manjugv manjugv approved these changes

Labels

Ready-for-Review