Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

a quick workaround for increasing the mpi task numbers on orion for ctest :: rrfs_3denvar_rdasens #788

Conversation

TingLei-daprediction
Copy link
Contributor

@TingLei-daprediction TingLei-daprediction commented Aug 30, 2024

It has been under investigation why the GSI would become idle in rrfs_3denvar_rdasens issue 766. Following discussions with @RussTreadon-NOAA discussion , this PR would provide a quick fix/workaround for rrfs_3denvar_rdasens running on orion by increasing the mpi task numbers running the test.
Update: it should be noted. the corresponding local branch had passed regression tests on orion before this PR was submitted.

…d the program hanging as reported in Issue 766
@RussTreadon-NOAA
Copy link
Contributor

@TingLei-daprediction , please reach out and find two peer reviewers for this PR.

Copy link

@SamuelDegelia-NOAA SamuelDegelia-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ctest rrfs_3denvar_rdasens completed successfully for me using this PR. It is still slower than Hercules but at least the test is working.

(base) orion-login-3[130] sdegelia$ ctest -R rrfs_3denvar_rdasens
Test project /work/noaa/wrfruc/sdegelia/GSI/build
    Start 3: rrfs_3denvar_rdasens
1/1 Test #3: rrfs_3denvar_rdasens .............   Passed  1146.50 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 1146.51 sec

@RussTreadon-NOAA
Copy link
Contributor

Thank you @SamuelDegelia-NOAA for running the rrfs_3denvar_rdasens on Orion. This is above and beyond what you need to do as a GSI Peer reviewer.

As the developer of this change and the author of this PR, @TingLei-daprediction should run and document ctest results. @TingLei-daprediction , please review GSI: How to Make Changes

@TingLei-NOAA
Copy link
Contributor

@SamuelDegelia-NOAA Thanks for your reviewing and ctests on your side. That will always be helpful for ctests by a "third-party".
@RussTreadon-NOAA For this PR specific for Orion, maybe I should also have added that I had done tests on orion before submitting this PR (by default in my opinion) and I don't think it is necessary to do tests on other machines unless the reviewer or the GSI managing team require. Thanks.

@RussTreadon-NOAA
Copy link
Contributor

@TingLei-daprediction , you are right. As this PR only alters one test on one machine, running rrfs_3denvar_rdasens on Orion is sufficient. I agree that it is useful for peer reviewers to run ctests. I usually do so. This, however, is not required. You, the author of these changes, should post results from your tests before peer reviewers are assigned.

@TingLei-daprediction
Copy link
Contributor Author

@RussTreadon-NOAA Thanks. I had updated the description for this PR. Please let me know if more details are needed.

@RussTreadon-NOAA
Copy link
Contributor

@TingLei-daprediction , what you added is sufficient for this PR. In the future please follow @SamuelDegelia-NOAA 's example and include the ctest output.

The ctest rrfs_3denvar_rdasens completed successfully for me using this PR. It is still slower than Hercules but at least the test is working.

(base) orion-login-3[130] sdegelia$ ctest -R rrfs_3denvar_rdasens
Test project /work/noaa/wrfruc/sdegelia/GSI/build
    Start 3: rrfs_3denvar_rdasens
1/1 Test #3: rrfs_3denvar_rdasens .............   Passed  1146.50 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 1146.51 sec

@TingLei-NOAA
Copy link
Contributor

@RussTreadon-NOAA Thanks!

Copy link
Contributor

@JingCheng-NOAA JingCheng-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since there are already two successful ctests (from Ting and Samuel) for this change on Orion, I have no issue to approve it.

@ShunLiu-NOAA ShunLiu-NOAA merged commit 8412e3f into NOAA-EMC:develop Sep 3, 2024
4 checks passed
DavidHuber-NOAA added a commit to DavidHuber-NOAA/GSI that referenced this pull request Sep 6, 2024
* origin/develop:
  Move to contrib spack-stack on Jet (NOAA-EMC#787)
  a quick workaround for increasing the mpi task numbers on orion for ctest :: rrfs_3denvar_rdasens  (NOAA-EMC#788)
  Recover the capability of handling model fields from operation gfs.v16.3 (NOAA-EMC#785)
  fix a bug in deter_sfc_gmi (NOAA-EMC#781)
  add safeguard to thompson_reff (NOAA-EMC#779)
  Fix incorrect usage of real(i_kind) in mg_input.f90  (NOAA-EMC#760)
  Transition to Thompson Microphysics for Microwave All-sky Assimilation (NOAA-EMC#743)
  Format changes for EUMETSAT metop-sg and CADS debug fix (NOAA-EMC#773)
  Update global_4denvar and global_enkf ctests to reflect GFS v17 (NOAA-EMC#774)
  fix for cris-fsr memory corruption (NOAA-EMC#767)
  Gnssrwnd1.0 (NOAA-EMC#747)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants