Add chgres_cube support for regional model (HRRR, RAP, and NAM) grib2 files and UFS_UTILS build support for Cheyenne, Stampede2, and Odin #146

JeffBeck-NOAA · 2020-09-10T03:30:23Z

This PR represents required UFS_UTILS changes for the SRW App release:

Adds support in chgres_cube for regional grib2 files for HRRR, RAP, and NAM models
Adds support in chgres_cube to interpolate from nine to four soil layers (initialize from nine-layer RUC-LSM RAP/HRRR data for CCPP SDFs using four-layer NOAH/NOAH MP)
Adds UFS_UTILS build/module support for Cheyenne, Stampede2, and Odin HPC systems, which are Tier 1 and Tier 2 platforms for the SRW App release
Modifies all module files to use "setenv" instead of "export" commands and modifies build_all.sh script to use "module use/load" instead of "source" to allow module files to be loaded at both build and run time
Adds logic to chgres_cube write_data.F90 file to output tile7 NetCDF files when "regional" flag is on
Adds support in chgres_cube to allow for either a dot or an underscore when reading in the mosaic file (C768.mosaic.halo4.nc or C768_mosaic.halo4.nc)

Both build and run time changes have been tested successfully on Hera and Jet (so far) using the SRW App community workflow. The build has also been tested on Cheyenne, Stampede2, and Odin, with full end-to-end workflow tests to commence shortly. Testing on WCOSS is pending.

The following developers have provided code for this PR: @LarissaReames-NOAA, @JeffBeck-NOAA, @ywangwof, @gsketefian, @JulieSchramm, and @mkavulich

… for sigio and sfcio

…te to find_package calls

ufs_release_1.0: cleanup work 2020/01/06

… for macOS with clang compilers

Catch unimplemented build method, add support for macOS compilers

As per conversation with Arun, since this affects the ufs_release_v1.0 branch only and not the main develop branch, I am going to merge this PR to allow the CIME people to progress with their work.

ufs_release_v1.0: re-add cmake module

Add macOS/clang support

….0' into ufs_release_v1.0

…UTILS into ufs_release_v1.0

Ufs release v1.0

… for cmake

…no, chgres_cube grib2 GFS data support); update submodule pointer for cmake

JeffBeck-NOAA · 2020-10-09T22:19:15Z

@JeffBeck-NOAA Under ./CMakeModules, your branch is pointing to 09ff76f. The authoritative 'develop' is pointing to a later version - 3eaddb8. When you merge from 'develop', are you getting an alert?

@GeorgeGayno-NOAA, sorry for the late reply. I was out this week on vacation. I pulled the latest develop branch code into my fork/branch, and it now points to 3eaddb8.

JeffBeck-NOAA · 2020-10-09T22:19:56Z

A note concerning the build modules. Changing them to be true modules, then doing a module load in the build script is perfectly fine. However, be aware those build modules are still being 'sourced' from the utility and regression test scripts. When I tried to create a C96 grid using ./driver_scripts/driver_grid_dell.sh, I got a warning about 'setenv': command not found. So, I made this change and the warning went away.

diff --git a/driver_scripts/driver_grid.dell.sh b/driver_scripts/driver_grid.dell.sh
index d2d6d08d..760c4415 100755
--- a/driver_scripts/driver_grid.dell.sh
+++ b/driver_scripts/driver_grid.dell.sh
@@ -59,7 +59,8 @@
#-----------------------------------------------------------------------

source ../sorc/machine-setup.sh > /dev/null 2>&1
-source ../modulefiles/build.$target
+module use ../modulefiles
+module load build.$target
module list

I can identify the scripts that needed to be updated and test them.

@GeorgeGayno-NOAA, sure! Please point me to the 'gitdiff' file and I'll be happy to incorporate those changes.

JeffBeck-NOAA · 2020-10-12T21:04:57Z

@GeorgeGayno-NOAA @LarissaReames-NOAA, the UFS_UTILS regression tests have passed for this PR on both Jet and Hera here:

Jet: /lfs4/BMC/wrfruc/beck/UFS_UTILS_Jeff/reg_tests/chgres_cube/regression.log
Hera: /scratch2/BMC/det/beck/FV3-LAM/UFS_UTILS_Jeff/reg_tests/chgres_cube/regression.log

This leaves us with Orion and WCOSS Dell and Cray to test.

GeorgeGayno-NOAA · 2020-10-13T14:09:17Z

A note concerning the build modules. Changing them to be true modules, then doing a module load in the build script is perfectly fine. However, be aware those build modules are still being 'sourced' from the utility and regression test scripts. When I tried to create a C96 grid using ./driver_scripts/driver_grid_dell.sh, I got a warning about 'setenv': command not found. So, I made this change and the warning went away.
diff --git a/driver_scripts/driver_grid.dell.sh b/driver_scripts/driver_grid.dell.sh
index d2d6d08d..760c4415 100755
--- a/driver_scripts/driver_grid.dell.sh
+++ b/driver_scripts/driver_grid.dell.sh
@@ -59,7 +59,8 @@
#-----------------------------------------------------------------------
source ../sorc/machine-setup.sh > /dev/null 2>&1
-source ../modulefiles/build.$target
+module use ../modulefiles
+module load build.$target
module list
I can identify the scripts that needed to be updated and test them.

@GeorgeGayno-NOAA, sure! Please point me to the 'gitdiff' file and I'll be happy to incorporate those changes.

Find the 'git diff' files on Hera: /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS.jeffb/$machine.diff.txt.

GeorgeGayno-NOAA · 2020-10-13T18:16:14Z

sorc/chgres_cube.fd/model_grid.F90

+ if (mod(npets,num_tiles_input_grid) /= 0) then
+   call error_handler("MUST RUN WITH A TASK COUNT THAT IS A MULTIPLE OF 6.", 1)
+ endif
+


In this case, the number of input tiles will always be one. So this error message will never be tripped.

@GeorgeGayno-NOAA, I merged in @LarissaReames-NOAA's change to remove this check in the latest commit below.

JeffBeck-NOAA · 2020-10-13T18:51:02Z

A note concerning the build modules. Changing them to be true modules, then doing a module load in the build script is perfectly fine. However, be aware those build modules are still being 'sourced' from the utility and regression test scripts. When I tried to create a C96 grid using ./driver_scripts/driver_grid_dell.sh, I got a warning about 'setenv': command not found. So, I made this change and the warning went away.
diff --git a/driver_scripts/driver_grid.dell.sh b/driver_scripts/driver_grid.dell.sh
index d2d6d08d..760c4415 100755
--- a/driver_scripts/driver_grid.dell.sh
+++ b/driver_scripts/driver_grid.dell.sh
@@ -59,7 +59,8 @@
#-----------------------------------------------------------------------
source ../sorc/machine-setup.sh > /dev/null 2>&1
-source ../modulefiles/build.$target
+module use ../modulefiles
+module load build.$target
module list
I can identify the scripts that needed to be updated and test them.

@GeorgeGayno-NOAA, sure! Please point me to the 'gitdiff' file and I'll be happy to incorporate those changes.

Find the 'git diff' files on Hera: /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS.jeffb/$machine.diff.txt.

Hi @GeorgeGayno-NOAA, I just pushed a commit to add the diff files (including a minor tweak to make sure the c96 regional regression test succeeds, since the baseline files are still "tile1" instead of "tile7").

…k-NOAA/UFS_UTILS into feature/regional_release

GeorgeGayno-NOAA · 2020-10-13T19:38:04Z

sorc/chgres_cube.fd/surface.F90

@@ -96,6 +98,9 @@ module surface
 type(esmf_field)                   :: terrain_from_input_grid
                                       ! terrain height interpolated
                                       ! from input grid
+ type(esmf_field)                   :: terrain_from_input_grid_land
+                                       ! terrain height interpolated
+                                       ! from input grid at all land points 



How is terrain_from_input_grid_land used? It is computed below. But I don't understand how it is used.

We were experiencing issues where isolated land ice points weren't ever able to obtain a proper terrain height, so this array is passed through during the search/replace of land ice points. Since soil temperature adjustment for terrain height is not turned off when points on the target grid have the default terrain height replacement value (default_value = -99999.9), providing data interpolated to "land points" which the input data originally deemed these land ice points to be (since it doesn't have land ice) allows for these points to have a proper terrain height. It looks like this use actually got coded out, but I'm putting it back in by providing the data from terrain_from_input_grid as an optional argument to the search function on line 2049 of surface.F90.

BenjaminBlake-NOAA · 2020-10-13T20:24:57Z

All of the regression tests for chgres cube passed on the WCOSS Dell after I added Jeff's tile1/tile7 fix to the c96 regional regression test script. I'm waiting on the results from the WCOSS Cray but I believe they will also be successful.

GeorgeGayno-NOAA · 2020-10-13T20:25:40Z

sorc/chgres_cube.fd/model_grid.F90

+       if (localpet==0) then
+         cmdline_msg = "wgrib2 "//trim(the_file)//" -d 1 -grid &> temp2.out"
+         call system(cmdline_msg)
+         open(4,file="temp2.out")


I thought this was replaced by a call to grb2_inq.

Ah, it looks like you found another use that I had forgotten about. Thanks for pointing that out. I should able to replace that relatively quickly.

BenjaminBlake-NOAA · 2020-10-13T20:32:43Z

The chgres cube regression tests all passed on the WCOSS Cray as well.

…ated use of terrain_from_input_grid_land to fill in missing target grid terrain height values at land ice points in the search/replace routine.

JeffBeck-NOAA · 2020-10-13T22:32:49Z

@GeorgeGayno-NOAA I just merged in @LarissaReames-NOAA's latest code fixes for the terrain and wgrib2 change.

GeorgeGayno-NOAA · 2020-10-14T14:46:47Z

There is something wrong with your latest commit (3393229). I see two 'mpif.h' in model_grid.F90. I thought they were all removed. Why are they reappearing? Also, an error message removed by Larissa has reappeared. We need to stop these fork-to-fork merges. The only merges should be from/to 'develop' of the authoritative repo.

…k-NOAA/UFS_UTILS into feature/regional_release

LarissaReames-NOAA · 2020-10-14T15:47:13Z

There is something wrong with your latest commit (3393229). I see two 'mpif.h' in model_grid.F90. I thought they were all removed. Why are they reappearing? Also, an error message removed by Larissa has reappeared. We need to stop these fork-to-fork merges. The only merges should be from/to 'develop' of the authoritative repo.

I'll now be merging straight to Jeff's branch. Those overwritten changes have now been put back in.

GeorgeGayno-NOAA · 2020-10-14T20:45:10Z

reg_tests/chgres_cube/driver.hera.sh_Jeff

@@ -0,0 +1,132 @@
+#!/bin/bash
+


This version should be removed.

@GeorgeGayno-NOAA Just removed that script.

…feature/regional_release

…Reames-NOAA/UFS_UTILS into feature/regional_release

JeffBeck-NOAA · 2020-10-14T23:42:16Z

@GeorgeGayno-NOAA, the develop branch of UFS_UTILS fails all regression tests on Orion with very small (1e-10 to 1e-17 differences), likely related to the baseline files either having been created on a different platform, or by using different compilers. However, I used the develop branch regression test results as the baseline for a test with the new code, and when I do that, the regression tests pass on Orion for this PR.

Therefore, all platforms (Orion, Hera, Jet, WCOSS Dell and Cray) now pass the regression tests for this PR.

GeorgeGayno-NOAA · 2020-10-15T12:43:40Z

@GeorgeGayno-NOAA, the develop branch of UFS_UTILS fails all regression tests on Orion with very small (1e-10 to 1e-17 differences), likely related to the baseline files either having been created on a different platform, or by using different compilers. However, I used the develop branch regression test results as the baseline for a test with the new code, and when I do that, the regression tests pass on Orion for this PR.

Therefore, all platforms (Orion, Hera, Jet, WCOSS Dell and Cray) now pass the regression tests for this PR.

Ok. I will do the merge today.

climbfuji and others added 30 commits January 6, 2020 08:21

Update .gitmodules, add branch = ufs_release_1.0 for cmake

836019c

Update .gitmodules for testing, code review and merging

a222cc1

Fix typos in several CMakeLists.txt files, update include directories…

61931f4

… for sigio and sfcio

Comment out LAPACK dependency in CMakeLists.txt, add REQUIRED attribu…

5a43265

…te to find_package calls

Remove REQUIRED attribute from find_package(ESMF)

f2d0f6e

Revert change to .gitmodules, update submodule pointer for cmake

11020a9

Merge pull request #41 from climbfuji/dom_cleanup_work_20191206

7dfe86b

ufs_release_1.0: cleanup work 2020/01/06

CMakeLists.txt: catch unimplemented build method, add missing support…

c20341a

… for macOS with clang compilers

Merge pull request #43 from climbfuji/esmf_guard_macos_support

f453ef9

Catch unimplemented build method, add support for macOS compilers

Re-factor, cleanup, and add export targets in CMake

d9c7e75

Re-enable find_package that was accidentally commented

e345dce

merge ufs_release_v1.0 branch to latest develop

82f279c

fix branch name in .gitmodules

c069063

Merge pull request #45 from jedwards4b/merge_to_develop

180737b

As per conversation with Arun, since this affects the ufs_release_v1.0 branch only and not the main develop branch, I am going to merge this PR to allow the CIME people to progress with their work.

Re-add submodule cmake

74b0fa5

Merge pull request #46 from climbfuji/readd_cmake_module

dc5bbce

ufs_release_v1.0: re-add cmake module

removed extraneous utils

0182118

Merge remote-tracking branch 'origin/release/ufs_release_v1.0' into tmp

3f6d0f9

Update required CMake version

0eab34d

Formatting

2c5a4d3

Add macOS/clang support for the third time

fc0a579

Merge pull request #1 from climbfuji/macos_clang_support

f8f7c65

Add macOS/clang support

Merge remote-tracking branch 'remotes/upstream/release/ufs_release_v1…

1a1ab75

….0' into ufs_release_v1.0

Merge branch 'ufs_release_v1.0' of https://github.com/kgerheiser/UFS_…

d1f1a25

…UTILS into ufs_release_v1.0

Merge pull request #49 from kgerheiser/ufs_release_v1.0

6d9e2c5

Ufs release v1.0

Update branch name for cmake submodule, update submodule pointer

e0fbeee

Update submodule pointer for cmake

f14769a

Bugfixes for CMakeLists.txt, update .gitmodules and submodule pointer…

1e1d1c8

… for cmake

Merge develop into release/public-v1 (changes: PR #48 from George Gay…

40c21bd

…no, chgres_cube grib2 GFS data support); update submodule pointer for cmake

Update submodule pointer for cmake

ef8f702

Change CMakeModules

d1ede40

GeorgeGayno-NOAA reviewed Oct 13, 2020

View reviewed changes

Changes to utility and regression test scripts.

c7d8bfd

LarissaReames-NOAA added 2 commits October 13, 2020 18:59

Merge branch 'feature/regional_release' of https://github.com/JeffBec…

d6dbccc

…k-NOAA/UFS_UTILS into feature/regional_release

Removed unused error message in model_grid.F90.

641fc98

GeorgeGayno-NOAA reviewed Oct 13, 2020

View reviewed changes

Removed additional use of wgrib2 executable in model_grid.F90. Reinst…

56283e2

…ated use of terrain_from_input_grid_land to fill in missing target grid terrain height values at land ice points in the search/replace routine.

Add CMakeModules

3393229

LarissaReames-NOAA added 2 commits October 14, 2020 15:34

Reapply changes that got over-written

47ba0e0

Merge branch 'feature/regional_release' of https://github.com/JeffBec…

4725146

…k-NOAA/UFS_UTILS into feature/regional_release

GeorgeGayno-NOAA reviewed Oct 14, 2020

View reviewed changes

Jeff Beck added 3 commits October 14, 2020 16:10

Remove personal shell script.

97ed123

Merge branch 'develop' of https://github.com/NOAA-EMC/UFS_UTILS into …

ce3220a

…feature/regional_release

Merge branch 'feature/regional_release' of https://github.com/Larissa…

63bfc92

…Reames-NOAA/UFS_UTILS into feature/regional_release

GeorgeGayno-NOAA merged commit f8dc806 into ufs-community:develop Oct 15, 2020

This was referenced Oct 16, 2020

Remove 'exports' from build modules #132

Closed

Merge regional chgres updates into authoritative repository #92

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add chgres_cube support for regional model (HRRR, RAP, and NAM) grib2 files and UFS_UTILS build support for Cheyenne, Stampede2, and Odin #146

Add chgres_cube support for regional model (HRRR, RAP, and NAM) grib2 files and UFS_UTILS build support for Cheyenne, Stampede2, and Odin #146

JeffBeck-NOAA commented Sep 10, 2020 •

edited

Loading

JeffBeck-NOAA commented Oct 9, 2020

JeffBeck-NOAA commented Oct 9, 2020

JeffBeck-NOAA commented Oct 12, 2020

GeorgeGayno-NOAA commented Oct 13, 2020

GeorgeGayno-NOAA Oct 13, 2020

JeffBeck-NOAA Oct 13, 2020 •

edited

Loading

JeffBeck-NOAA commented Oct 13, 2020

GeorgeGayno-NOAA Oct 13, 2020

LarissaReames-NOAA Oct 13, 2020

BenjaminBlake-NOAA commented Oct 13, 2020

GeorgeGayno-NOAA Oct 13, 2020

LarissaReames-NOAA Oct 13, 2020

BenjaminBlake-NOAA commented Oct 13, 2020

JeffBeck-NOAA commented Oct 13, 2020

GeorgeGayno-NOAA commented Oct 14, 2020

LarissaReames-NOAA commented Oct 14, 2020

GeorgeGayno-NOAA Oct 14, 2020

JeffBeck-NOAA Oct 14, 2020

JeffBeck-NOAA commented Oct 14, 2020

GeorgeGayno-NOAA commented Oct 15, 2020

Add chgres_cube support for regional model (HRRR, RAP, and NAM) grib2 files and UFS_UTILS build support for Cheyenne, Stampede2, and Odin #146

Add chgres_cube support for regional model (HRRR, RAP, and NAM) grib2 files and UFS_UTILS build support for Cheyenne, Stampede2, and Odin #146

Conversation

JeffBeck-NOAA commented Sep 10, 2020 • edited Loading

JeffBeck-NOAA commented Oct 9, 2020

JeffBeck-NOAA commented Oct 9, 2020

JeffBeck-NOAA commented Oct 12, 2020

GeorgeGayno-NOAA commented Oct 13, 2020

GeorgeGayno-NOAA Oct 13, 2020

Choose a reason for hiding this comment

JeffBeck-NOAA Oct 13, 2020 • edited Loading

Choose a reason for hiding this comment

JeffBeck-NOAA commented Oct 13, 2020

GeorgeGayno-NOAA Oct 13, 2020

Choose a reason for hiding this comment

LarissaReames-NOAA Oct 13, 2020

Choose a reason for hiding this comment

BenjaminBlake-NOAA commented Oct 13, 2020

GeorgeGayno-NOAA Oct 13, 2020

Choose a reason for hiding this comment

LarissaReames-NOAA Oct 13, 2020

Choose a reason for hiding this comment

BenjaminBlake-NOAA commented Oct 13, 2020

JeffBeck-NOAA commented Oct 13, 2020

GeorgeGayno-NOAA commented Oct 14, 2020

LarissaReames-NOAA commented Oct 14, 2020

GeorgeGayno-NOAA Oct 14, 2020

Choose a reason for hiding this comment

JeffBeck-NOAA Oct 14, 2020

Choose a reason for hiding this comment

JeffBeck-NOAA commented Oct 14, 2020

GeorgeGayno-NOAA commented Oct 15, 2020

JeffBeck-NOAA commented Sep 10, 2020 •

edited

Loading

JeffBeck-NOAA Oct 13, 2020 •

edited

Loading