Fix for inefficient iceberg restarts #1460
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
register_unlimited_compressed_axis. Here, the iceberg dimension length for each PE was being sent to all other PEs, an operation inefficient enough to cause runs with a large number of PEs to appear to stall until they timed out. However, these local dimension lengths actually only need to be gathered on the I/O root PE. Making this change fixed the issue.
alex-huth
requested review from
thomas-robinson,
bensonr and
rem1776
as code owners
rem1776
approved these changes
May 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changed registration of axes during iceberg restarts so that dimension lengths from each PE are gathered to only the I/O root PE, replacing the original approach of gathering to all PEs. The original approach was inefficient enough to stall runs with a large number of PEs until they timed out, and this PR fixes this issue.
How Has This Been Tested?
intel-classic/2022.0.2. Confirmed that the restart files produced with/without this commit are identical
Checklist:
make distcheckpasses