For spack-stack@1.5.0 / GSI: add bufr@11..7.1 to unified environment (no, no longer needed) and switch to gsi-ncdiag@1.1.1

[climbfuji]

Is your feature request related to a problem? Please describe.
Based on discussions in #643 and #687 we need to install bufr@11.7.1 (or 11.7.0 - I assume the hotfix is better? @DavidHuber-NOAA) in addition to the default bufr@12.0.0 in the spack-stack@1.5.0 unified environment for GSI. Further, we need to make gsi-ncdiag@1.1.1 the default if it isn't already.

Once that's done and rolled out on Hercules, we need to ask @jswhit to retest and see if this (and/or the compiler update we did on Hercules) fixes #643.

Describe the solution you'd like
See above

Additional context
See linked discussions

[DavidHuber-NOAA]

I haven't seen/heard any conversations about bringing bufr 11.7.1 into the GSI and I'm not familiar with the hotfixes that went into it from 11.7.0. It would require testing for validation. Any suggestions/preferences @RussTreadon-NOAA or @mark-a-potts?

[RussTreadon-NOAA]

This is the first I heard of bufr/11.7.1. Seems the next major release is bufr/12.0.0. The GSI has been tested and ctests run with bufr/12.0.0.

[DavidHuber-NOAA]

Ah, alright, so it sounds like we could move to just bufr/12.0.0 for spack-stack/1.5.0. We would need gsi-ncdiag/1.1.1 added though.

[climbfuji]

That's a lot easier then! Good to know that we can get by with just bufr@12 - thanks!

[climbfuji]

Turns out nothing needs to be done. gsi-ncdiag in unified-env/skylab-env is already at 1.1.1, bufr is at 12.0.0, and there are two separate issues that deal with updating the ufs-weather-model-static and ufs-srw-app-* templates for the release.

[climbfuji]

@DavidHuber-NOAA @RussTreadon-NOAA How can I test GSI with bufr@12?

I installed the spack-stack-1.5.0 release candidate on Hercules with bufr@12 only as discussed above, but the GSI dev branch from today Sep 10 (from noaa-emc github) still wants bufr::bufr_d. Are there any special flags I need to set?

> module li
Currently Loaded Modules:
  1) ecflow/5.8.4                      6) stack-intel-oneapi-mpi/2021.9.0  11) bacio/2.4.1         16) zstd/1.5.2      21) netcdf-fortran/4.6.0  26) ncio/1.1.2    31) wrf-io/1.2.0
  2) mysql/8.0.31                      7) zlib/1.2.13                      12) bufr/12.0.0         17) c-blosc/1.21.4  22) crtm/2.4.0            27) w3emc/2.9.2   32) gsi-env/1.0.0
  3) intel-oneapi-compilers/2023.1.0   8) sqlite/3.39.4                    13) crtm-fix/2.4.0_emc  18) curl/8.1.2      23) gsi-ncdiag/1.1.1      28) nemsio/2.5.4  33) openblas/0.3.19
  4) stack-intel/2021.9.0              9) python/3.10.8                    14) git-lfs/3.1.2       19) hdf5/1.14.0     24) sp/2.3.3              29) sfcio/1.4.1
  5) intel-oneapi-mpi/2021.9.0        10) stack-python/3.10.8              15) snappy/1.1.10       20) netcdf-c/4.9.2  25) ip/3.3.3              30) sigio/2.3.2

> cmake -DCMAKE_INSTALL_PREFIX=../install ../gsi-dev-20230910 2>&1 | tee log.cmake
...
CMake Error at src/gsi/CMakeLists.txt:133 (add_library):
  Target "gsi_fortran_obj" links to target "bufr::bufr_d" but the target was
  not found.  Perhaps a find_package() call is missing for an IMPORTED
  target, or an ALIAS target is missing?


CMake Error at src/enkf/CMakeLists.txt:97 (add_library):
  Target "enkf_fortran_obj" links to target "bufr::bufr_d" but the target was
  not found.  Perhaps a find_package() call is missing for an IMPORTED
  target, or an ALIAS target is missing?

[RussTreadon-NOAA]

@climbfuji , I do not have any version of GSI committed to any repos which can run with bufr/12. GSI issue #589 describes changes made to get the GSI code to work with bufr/12. Click here for required modifications to read_prepbufr.f90.

I tested these changes on Hera in /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/bufr12. This was an old copy of develop with required bufr/12 changes added. I updated this directory this morning (9/11) with the current head of develop, recompile, and reran the global_3dvar ctest. The test ran to completion.

Hera(hfe02):/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/bufr12/build$ ctest -R global_3dvar
Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/bufr12/build
    Start 1: global_3dvar
1/1 Test #1: global_3dvar .....................***Failed  1971.92 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 1971.93 sec

The following tests FAILED:
          1 - global_3dvar (Failed)
Errors while running CTest
Output from these tests are in: /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/bufr12/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

The failure is due to

The runtime for global_3dvar_hiproc_updat is 359.155741 seconds.  This has exceeded maximum allowable threshold time of 353.483517 seconds,
resulting in Failure of timethresh2 the regression test.

Indeed gsi.x built with bufr/12 runs slower than gsi.x built with bufr/11.7

global_3dvar_hiproc_contrl/stdout:The total amount of wall time                        = 321.348652
global_3dvar_hiproc_updat/stdout:The total amount of wall time                        = 359.155741
global_3dvar_loproc_contrl/stdout:The total amount of wall time                        = 471.704760
global_3dvar_loproc_updat/stdout:The total amount of wall time                        = 516.706617

I do not know if this is known or expected behavior with bufr/12.

Here are the modified files in /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/bufr12

        modified:   modulefiles/gsi_common.lua
        modified:   src/gsi/CMakeLists.txt
        modified:   src/gsi/read_prepbufr.f90
        modified:   ush/sub_hera

A brief description of the local modifications follows:

• modulefiles/gsi_common.lua was modified to use a July 20, 2023 copy of bufr/12 from @DavidHuber-NOAA
• src/gsi/CMakeLists.txt links library bufr_4 instead of bufr_d
• src/gsi/read_prepbufr.f90 uses an integer argument in the call to ufbqcd
• ush/sub_hera contains hardwired changes to load the appropriate modules when running the bufr12 or unaltered develop builds

[climbfuji]

Thanks for the detailed explanation. I am still not sure what we are supposed to do, though. Release spack-stack-1.5.0 with bufr@12.0.0 only, as it was suggested above (#738 (comment)) or make a last-minute addition of bufr@11.7.1/0. We are rolling out the release starting today.

[RussTreadon-NOAA]

The GSI build currently uses bufr/11.7.0. The GDASApp build uses bufr/11.7.1. It seems both eventually need to move to bufr/12.0.0. If true, building spack-stack/1.5.0 with bufr/12.0.0 sounds reasonable. That said, I'm not sure what other apps are or will be built with spack-stack/1.5.0. Might any of the potential users of spack-stack/1.5.0 prefer or require bufr/11.7.1? I don't know.

[climbfuji]

Alright, let's only have 12 in spack-stack-1.5.0, which will require updating GSI when it moves to spack-stack. The ufs-weather-model and ufs-srw-app will be using spack-stack-1.5.0. At JCSDA, we also us it for UFS_UTILS.