Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add specific communicator for neighborhood communication #1588

Open
wants to merge 12 commits into
base: index-map-pgm
Choose a base branch
from
Open

Add specific communicator for neighborhood communication #1588

wants to merge 12 commits into from

Conversation

MarcelKoch
Copy link
Member

@MarcelKoch MarcelKoch commented Apr 4, 2024 •
edited
Loading

This PR adds a communicator that only handles neighborhood all-to-all communication. It implements the new interface collective_communicator, which provides different implementations for a selected set of collective mpi routines. Currently, this only includes the non-blocking all-to-all.

The communication uses a fixed pattern, i.e. the send/recv sizes are fixed when the neighborhood communicator is constructed. I would have liked to decouple that, but this would require some knowledge of how the sizes are stored at the interface level. If someone has an idea for that, please let me know.

This is the first part of splitting up #1546.

The neighborhood all-to-all has a bug in OpenMPI < v4.1.0, which makes it necessary to disable the neighborhood communicator in this case. As replacement, there is also a dense all-to-all communicator.

Todo:

  • documentation

PR Stack:

@MarcelKoch MarcelKoch self-assigned this Apr 4, 2024
@ginkgo-bot ginkgo-bot added reg:build This is related to the build system. reg:testing This is related to testing. mod:core This is related to the core module. labels Apr 4, 2024
@MarcelKoch MarcelKoch requested a review from pratikvn April 4, 2024 10:41
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from b42ab92 to 8f104fd Compare April 4, 2024 11:00
@MarcelKoch MarcelKoch modified the milestone: Ginkgo 1.8.0 Apr 5, 2024
@MarcelKoch MarcelKoch requested a review from upsj April 19, 2024 09:19
@MarcelKoch MarcelKoch mentioned this pull request Apr 19, 2024
Closed
7 tasks
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from 8f104fd to a0824a8 Compare April 19, 2024 14:39
This was referenced Apr 22, 2024
Merged
Merged
Merged
Merged
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from a0824a8 to 8ad3f2f Compare April 22, 2024 11:11
@MarcelKoch MarcelKoch mentioned this pull request Apr 30, 2024
Open
6 tasks
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from 8ad3f2f to 26678b3 Compare April 30, 2024 13:41
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from 26678b3 to 006d67d Compare April 30, 2024 15:20
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from 006d67d to b295b11 Compare May 2, 2024 10:04
pratikvn
pratikvn requested changes Jul 23, 2024
View reviewed changes
Copy link
Member

@pratikvn pratikvn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good. Some missing tests and doc updates needed

core/test/mpi/distributed/dense_communicator.cpp Show resolved Hide resolved
core/distributed/dense_communicator.cpp Show resolved Hide resolved
include/ginkgo/core/distributed/dense_communicator.hpp Outdated Show resolved Hide resolved
@MarcelKoch MarcelKoch requested review from upsj and removed request for upsj August 27, 2024 12:05
upsj
upsj reviewed Oct 18, 2024
View reviewed changes
Copy link
Member

@upsj upsj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass, mostly interface and implementation

  • What is the moved-from state of a Communicator? Should it match that of an MPI communicator wrapper, and be MPI_NULL, or preserve the MPI communicator?

include/ginkgo/core/distributed/collective_communicator.hpp Outdated Show resolved Hide resolved
include/ginkgo/core/distributed/collective_communicator.hpp Outdated Show resolved Hide resolved
core/distributed/dense_communicator.cpp Outdated Show resolved Hide resolved
core/distributed/dense_communicator.cpp Outdated Show resolved Hide resolved
core/distributed/neighborhood_communicator.cpp Outdated Show resolved Hide resolved
@MarcelKoch
Copy link
Member Author

@upsj I think the moved-from that should be using MPI_COMM_NULL as communicator. Then trying to launch a communication will fail quite badly.

@MarcelKoch MarcelKoch force-pushed the index-map-pgm branch 2 times, most recently from 604a6e9 to ba0982e Compare October 23, 2024 15:13
MarcelKoch and others added 11 commits October 30, 2024 15:38
@MarcelKoch
Make comm_index_type avaiable in the mpi namespace
eb27dd5
@MarcelKoch
create owning communicator
4a18dd9
@MarcelKoch
[imap] create variant type of supported index maps
ac60daf
@MarcelKoch @pratikvn
[coll-comm] adds interface for collective communicator
e083e23 
Co-authored-by: Pratik Nayak <pratik.nayak@kit.edu>
@MarcelKoch
[coll-comm] add dense communication
9a83b32
@MarcelKoch @pratikvn
[coll-comm] adds neighborhood implementation of collective communicator
382fcf3 
Co-authored-by: Pratik Nayak <pratik.nayak@kit.edu>
@MarcelKoch
[coll-comm] disable neighbor comm if open-mpi version < 4.1.0
eba3a46
@MarcelKoch
[mpi] define moved-from state for communicator
a89a452
@MarcelKoch
[mpi] add explicit comparison for identical and congruent
f326ec3
@MarcelKoch @pratikvn @upsj
[coll-comm] review update:
785a865 
- fix include guards
- update docs
- implement copy/move constructors/assignment with tests
- add equality test for collective communicators (needed for testing)
- always enable neighborhood comm, just throw if openmpi is too old
- define moved-from state as MPI_COMM_NULL + empty sizes/offsets
- remove unnecessary namespace
- make virtual function protected

Co-authored-by: Pratik Nayak <pratik.nayak@kit.edu>
Co-authored-by: Tobias Ribizel <mail@ribizel.de>
@MarcelKoch
[coll-comm] remove envelop-based constructors
e7d32a1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@upsj upsj upsj left review comments

@pratikvn pratikvn pratikvn approved these changes

Assignees

@MarcelKoch MarcelKoch

Labels
1:ST:ready-for-review This PR is ready for review mod:core This is related to the core module. reg:build This is related to the build system. reg:testing This is related to testing.
Projects
None yet
Milestone
Ginkgo 1.9.0
Development

Successfully merging this pull request may close these issues.
4 participants
@MarcelKoch @pratikvn @upsj @ginkgo-bot