Add RRFS grib2 files to officially supported input sources of chgres_cube

[LarissaReames-NOAA]

The SRW App team have requested that we officially support RRFS grib2 data from the Amazon S3 buckets as input data as part of preparation for RRFS pre-operational support. Hopefully this shouldn't be too much work, but it may involve tracking down some discrepancies in grib2 table codes.

[GeorgeGayno-NOAA]

Someone at EMC requested this capability two years ago. (See #660). I started the code updates, but then found that the RRFS GRIB2 file was too big for our G2 library. I believe that G2 limitation was fixed.

[LarissaReames-NOAA]

Should we close this issue and go back to the work you did there and see if we can get a clean PR for it now that the G2 issue was fixed?

[JeffBeck-NOAA]

We can close this issue and add a comment in #660 mentioning resolution of the G2 issue and explaining a goal of getting this feature functional for the v2.2.0 release of the SRW App.

[edwardhartnett]

Some of the NCEPLIBS-g2 issues with 2 GB have been fixed, but not all. The issue that was fixed in g2-3.4.7 allows the (soon to be released) grib_utils utility degrib2 to read > 2 GB files.

But the problem you are having is with index files. There is a fundamental limitation with the current index file format which must be overcome. (It uses a 32-bit int for file offsets, this will not work beyond the 2 GB boundary). This will be addressed in the next release of NCEPLIBS-g2. (See NOAA-EMC/NCEPLIBS-g2#550).

Sorry for the long delay in GRIB library work. Our previous GRIB programmers retired/moved on and it's taken time for me to come up to speed on the code and to deal with the backlog of fixes needed.

[BenjaminBlake-NOAA]

The operational RRFS will also need this capability in order to run the fire weather nest initialized off of RRFS rotated lat-lon grib2 data.

[GeorgeGayno-NOAA]

@LarissaReames-NOAA and @BenjaminBlake-NOAA what are your deadlines for this capability?

[JeffBeck-NOAA]

@GeorgeGayno-NOAA, there was also an interest to have this capability available for the next SRW App release so that users could initialize their own simulations with RRFS data. Therefore, while we'll miss this next SRW App release, an "ASAP" deadline is appropriate, since it's necessary for the general community, fire weather (as @BenjaminBlake-NOAA mentioned), and for the 3D-RTMA (Annette Gibbs at EMC).

[edwardhartnett]

I am hearing that getting NCEPLIBS-g2 working with > 2 GB GRIB2 files is a priority, and I'll am working on it now.

Unfortunately it requires some major changes in g2 and grib_utils, and that will take some time.

I hope to have new releases of g2 and grib_utils which handle > 2 GB files sometime before the end of the year.

[LarissaReames-NOAA]

Now that the new g2 library is working, and #901 is on its way to being resolved, I'm working on supporting the full RRFS rotated lat Lon files. I'm running in to an issue of lots of missing data around the edges due to differences between the output and computational domains. See this plot of surface pressure

At these points, I'm getting divide by zero issues for some computations. And even if I wasn't we don't have any way to ensure that the user's target grid doesn't extend in to this missing data.

@GeorgeGayno-NOAA @JeffBeck-NOAA Do you have any thoughts on how to handle this? Do we fill the bad points with nonsense values just to get the computations to complete but perform some sort of check to make sure the target grid falls within the original computation grid (i.e., doesn't extend in to the missing points)? Or maybe we create some sort of good-points array that constrains the computation loops to only points inside the computational domain along with some sort of check about the user's target domain? Also, how might that target domain check work? There's a lot to think about here and I'd like your thoughts.

[GeorgeGayno-NOAA]

Now that the new g2 library is working, and #901 is on its way to being resolved, I'm working on supporting the full RRFS rotated lat Lon files. I'm running in to an issue of lots of missing data around the edges due to differences between the output and computational domains. See this plot of surface pressure

At these points, I'm getting divide by zero issues for some computations. And even if I wasn't we don't have any way to ensure that the user's target grid doesn't extend in to this missing data.

@GeorgeGayno-NOAA @JeffBeck-NOAA Do you have any thoughts on how to handle this? Do we fill the bad points with nonsense values just to get the computations to complete but perform some sort of check to make sure the target grid falls within the original computation grid (i.e., doesn't extend in to the missing points)? Or maybe we create some sort of good-points array that constrains the computation loops to only points inside the computational domain along with some sort of check about the user's target domain? Also, how might that target domain check work? There's a lot to think about here and I'd like your thoughts.

I seem to recall that with the right settings, the ESMF regridding routine will return an 'out of domain' code? You might want to read through the documentation or ask the ESMF team for advice. They are very responsive.

[natalie-perlin]

@LarissaReames-NOAA - has this RRFS grid capability been fully added?

Running into with .sfc_data.tile7.halo0.nc file prepared by chgres_cube, errors with reading this file during the forecast phase. The PR is open in SRW repo PR-1089:
ufs-community/ufs-srweather-app#1089

A tag (57bd832) of the UFS_UTILS checked out as a part of the SRW, did not allow for RRFS option. When this RRFS options was changed manually in ./UFS_UTILS/sorc/chgres_cube.fd/program_setup.F90, and other options in SRW needed for the RRFS ics/lbcs were set similar/identical to those for HRRR, the test passes successfully. When the updated UFS_UTILS tag was used (1dac855) that included options for RRFS, the error resulted during runtime.

The difference appeared to be in *.sfc_data.tile7.halo0.nc that had variations in its content from the expected by the SRW forecast phase.
The input files for the RRFS were retrieved from the following location,
https://noaa-rrfs-pds.s3.amazonaws.com/index.html#rrfs_a/rrfs_a.20230501/00/control/
as suggested at https://registry.opendata.aws/noaa-rrfs/
This AWS bucket also contains sfc_data.nc file under INPUT. Could it be that this needs to be integrated into SRW data when RRFS option is used by chgres_cube? What is the best way to find out the changes needed to read a new format?

[GeorgeGayno-NOAA]

@LarissaReames-NOAA - has this RRFS grid capability been fully added?

Running into with .sfc_data.tile7.halo0.nc file prepared by chgres_cube, errors with reading this file during the forecast phase. The PR is open in SRW repo PR-1089: ufs-community/ufs-srweather-app#1089

A tag (57bd832) of the UFS_UTILS checked out as a part of the SRW, did not allow for RRFS option. When this RRFS options was changed manually in ./UFS_UTILS/sorc/chgres_cube.fd/program_setup.F90, and other options in SRW needed for the RRFS ics/lbcs were set similar/identical to those for HRRR, the test passes successfully. When the updated UFS_UTILS tag was used (1dac855) that included options for RRFS, the error resulted during runtime.

chgres_cube was updated at 7addff5 for fractional grids. That required a slight format change (v2) in the surface coldstart file. At that time, the UFS was updated to read either the v2 or previous version of the surface file. What tag of UFS are you using?

The difference appeared to be in *.sfc_data.tile7.halo0.nc that had variations in its content from the expected by the SRW forecast phase. The input files for the RRFS were retrieved from the following location, https://noaa-rrfs-pds.s3.amazonaws.com/index.html#rrfs_a/rrfs_a.20230501/00/control/ as suggested at https://registry.opendata.aws/noaa-rrfs/ This AWS bucket also contains sfc_data.nc file under INPUT. Could it be that this needs to be integrated into SRW data when RRFS option is used by chgres_cube? What is the best way to find out the changes needed to read a new format?

[LarissaReames-NOAA]

@LarissaReames-NOAA - has this RRFS grid capability been fully added?

I have a branch that's fully capable of processing RRFS data with a newer version of the g2 library. However, the g2 version in spack-stack has not been updated to a version with the new large file capabilities required to process the full RRFS grid files. Should we put in a request to have this library updated? There are a couple of official g2 releases that have the large file capability.

[natalie-perlin]

@LarissaReames-NOAA - has this RRFS grid capability been fully added?
Running into with .sfc_data.tile7.halo0.nc file prepared by chgres_cube, errors with reading this file during the forecast phase. The PR is open in SRW repo PR-1089: ufs-community/ufs-srweather-app#1089
A tag (57bd832) of the UFS_UTILS checked out as a part of the SRW, did not allow for RRFS option. When this RRFS options was changed manually in ./UFS_UTILS/sorc/chgres_cube.fd/program_setup.F90, and other options in SRW needed for the RRFS ics/lbcs were set similar/identical to those for HRRR, the test passes successfully. When the updated UFS_UTILS tag was used (1dac855) that included options for RRFS, the error resulted during runtime.

chgres_cube was updated at 7addff5 for fractional grids. That required a slight format change (v2) in the surface coldstart file. At that time, the UFS was updated to read either the v2 or previous version of the surface file. What tag of UFS are you using?

The UFS tag 1c6b4d4 from May 16, 2024, was used.

Also, I'm using RRFS data for 20230501 timestamp. Is more recent RRFS model data needed to be used to make it work? Is the updated version of UFS (tag??) needed to read the RRFS model data correctly?

[natalie-perlin]

@LarissaReames-NOAA - has this RRFS grid capability been fully added?

I have a branch that's fully capable of processing RRFS data with a newer version of the g2 >library. However, the g2 version in spack-stack has not been updated to a version with the >new large file capabilities required to process the full RRFS grid files. Should we put in a >request to have this library updated? There are a couple of official g2 releases that have >the large file capability.

Thank you @LarissaReames-NOAA! Is it a branch of UFS_UTILS or UFS_WeatherModel?
What version of g2 is needed? There are a couple of higher versions of spack-stack (1.6.0 , 1.7.0) available that could be tested.

[LarissaReames-NOAA]

@LarissaReames-NOAA - has this RRFS grid capability been fully added?

I have a branch that's fully capable of processing RRFS data with a newer version of the g2 >library. However, the g2 version in spack-stack has not been updated to a version with the >new large file capabilities required to process the full RRFS grid files. Should we put in a >request to have this library updated? There are a couple of official g2 releases that have >the large file capability.

Thank you @LarissaReames-NOAA! Is it a branch of UFS_UTILS or UFS_WeatherModel? What version of g2 is needed? There are a couple of higher versions of spack-stack (1.6.0 , 1.7.0) available that could be tested.

This is a branch of UFS_UTILS that should allow you to create ICs and LBCs from the full RRFS files. I haven't created a PR as I wasn't sure if the newer g2 version was available yet in any spack. In my testing I've used 3.4.7 and I believe that's the earliest release that has the large file updates needed.

[natalie-perlin]

This is a branch of UFS_UTILS that should allow you to create ICs and LBCs from the full RRFS files. I haven't created a PR as I wasn't sure if the newer g2 version was available yet in any spack. In my testing I've used 3.4.7 and I believe that's the earliest release that has the large file updates needed.

The spack-stack-1.6.0 that I use has g2/3.4.5. It seem to work OK processing RRFS files, when I use files such as rrfs.t00z.prslev.f003.conus_3km.grib2.
Is it the right type of files to use, or are there different files that are expected?

Are there corresponding changes in expected format need to be done for the UFS-weather-model side for reading the RRFS data prepared by the UFS_UTILS?..

[LarissaReames-NOAA]

This is a branch of UFS_UTILS that should allow you to create ICs and LBCs from the full RRFS files. I haven't created a PR as I wasn't sure if the newer g2 version was available yet in any spack. In my testing I've used 3.4.7 and I believe that's the earliest release that has the large file updates needed.

The spack-stack-1.6.0 that I use has g2/3.4.5. It seem to work OK processing RRFS files, when I use files such as rrfs.t00z.prslev.f003.conus_3km.grib2. Is it the right type of files to use, or are there different files that are expected?

Are there corresponding changes in expected format need to be done for the UFS-weather-model side for reading the RRFS data prepared by the UFS_UTILS?..

Yeah that's a completely different kind of file. That data has been cropped and interpolated to a CONUS Lambert conformal grid, so it could probably honestly be used just as you use HRRR data for initialization and with the current develop branch of UFS_UTILS. The files I'm talking about are nearly 10GB in size and are from the full North American RAP domain and are what the changes in my branch would target. They have the format rrfs.t00z.natlev.f000.grib2.

[natalie-perlin]

Thank you, @LarissaReames-NOAA! That is important topic to converge on. I'm pulling RRFS data from an official RRFS-prototype data source mentioned in this site:
https://registry.opendata.aws/noaa-rrfs/
It has data available RRFS_A runs over CONUS domain, with the output on Lambert Conic Conformal grid at 3km grid spacing,
and has two data formats:
rrfs.t00z.natlev.f003.conus_3km.grib2 (about 1.4 GB)
and
rrfs.t00z.prslev.f003.conus_3km.grib2 (about 909MB)

This is RRFS data source that could be used to do automatic retrieval of the RRFS data. It is the source I was suggested to use by Christina Holt for RRFS - SRW integration work

What is the data you are using, and where could it be found?
Do we need another option for using RRFS data if retrieved from https://registry.opendata.aws/noaa-rrfs/?

[LarissaReames-NOAA]

@natalie-perlin That's the same place I've gotten mine from. If you dig around further there are other files without the conus_3km tag on them, including files on pressure levels as well like this one: https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/rrfs_a.20240610/00/control/rrfs.t00z.prslev.f000.grib2

I think it's important that we have these full RRFS files working if we want to be able to initialize larger domains, including a full CONUS run, from RRFS data.

[natalie-perlin]

Oh,

@natalie-perlin That's the same place I've gotten mine from. If you dig around further there are other files without the conus_3km tag on them, including files on pressure levels as well like this one: https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/rrfs_a.20240610/00/control/rrfs.t00z.prslev.f000.grib2

I think it's important that we have these full RRFS files working if we want to be able to initialize larger domains, including a full CONUS run, from RRFS data.

Thank you! Found these files. The seem to be 6GB in size.

Are there any changes expected to be on the UFS-WM side to read the UFS-UTILS processed files?..

[LarissaReames-NOAA]

Oh,

@natalie-perlin That's the same place I've gotten mine from. If you dig around further there are other files without the conus_3km tag on them, including files on pressure levels as well like this one: https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/rrfs_a.20240610/00/control/rrfs.t00z.prslev.f000.grib2
I think it's important that we have these full RRFS files working if we want to be able to initialize larger domains, including a full CONUS run, from RRFS data.

Thank you! Found these files. The seem to be 6GB in size.

Are there any changes expected to be on the UFS-WM side to read the UFS-UTILS processed files?..

On the weather model side, no it shouldn't matter as the data should look to the weather model like anything else coming from UFS_UTILS. Obviously the App/workflow would need updates.

@JeffBeck-NOAA Would you say that's correct or have I missed something?

[JeffBeck-NOAA]

Yep, nothing should have to change in the ufs-weather-model code, just the necessary changes in UFS-UTiLS and the workflow.

[natalie-perlin]

Awesome, thank you, @LarissaReames-NOAA and @JeffBeck-NOAA! Testing it with larger files.

And yes, the SRW workflow is adapted to use RRFS option, find filenames, to use variable-mapping table GSDphys_var_map.txt (as is used for RAP and HRRR), and also to use 9-level soil data.

[natalie-perlin]

@LarissaReames-NOAA @JeffBeck-NOAA -
SRW tasks make_ics and make_lbcs fail with the error that appears to be in reading surface pressure record:
...

- CALL FieldScatter FOR INPUT U-WIND.
 - CALL FieldScatter FOR INPUT V-WIND.
 - READ SURFACE PRESSURE.
 - FATAL ERROR: READING SURFACE PRESSURE RECORD.
 - IOSTAT IS:           99
Abort(999) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 0
/lustre/SRW/srw-upd_ufs_utils/ush/bash_utils/print_msg.sh: line 192: BASH_SOURCE[1]: unbound variable
FATAL ERROR:
...

Are there any suggestions on what may need attention?..

This is what config file for the ICS task look like (fort.41 in the tmp_MAKE_ICS work directory)

&config
    fix_dir_target_grid = '/lustre/SRW/expt_dirs/RRFS_CONUScompact_25km_UFS_UTILS/fix_lam'
    mosaic_file_target_grid = '/lustre/SRW/expt_dirs/RRFS_CONUScompact_25km_UFS_UTILS/fix_lam/C403_mosaic.halo4.nc'
    orog_dir_target_grid = '/lustre/SRW/expt_dirs/RRFS_CONUScompact_25km_UFS_UTILS/fix_lam'
    orog_files_target_grid = 'C403_oro_data.tile7.halo4.nc'
    vcoord_file_target_grid = '/lustre/SRW/expt_dirs/RRFS_CONUScompact_25km_UFS_UTILS/fix_am/global_hyblev.l65.txt'
    varmap_file = '/lustre/SRW/srw-upd_ufs_utils/parm/ufs_utils/varmap_tables/GSDphys_var_map.txt'
    data_dir_input_grid = '/lustre/SRW/expt_dirs/RRFS_CONUScompact_25km_UFS_UTILS/2024060517/RRFS/for_ICS/'
    atm_files_input_grid = ,
    sfc_files_input_grid = ,
    grib2_file_input_grid = 'rrfs.t17z.prslev.f000.grib2'
    cycle_mon = 6
    cycle_day = 5
    cycle_hour = 17
    convert_atm = .true.
    convert_sfc = .true.
    convert_nst = .false.
    regional = 1
    halo_bndy = 4
    halo_blend = 10
    input_type = 'grib2'
    external_model = 'RRFS'
    tracers_input = ''
    tracers = ''
    nsoill_out = 9
    geogrid_file_input_grid = '/contrib/EPIC/UFS_SRW_data/develop/fix/fix_am/geo_em.d01.nc_HRRRX'
    vgtyp_from_climo = .true.
    sotyp_from_climo = .true.
    vgfrc_from_climo = .true.
    minmax_vgfrc_from_climo = .true.
    lai_from_climo = .true.
    tg3_from_soil = .true.
    thomp_mp_climo_file = ,
/

[LarissaReames-NOAA]

@natalie-perlin Are you using the develop branch here? You're not going to be able to use the full RRFS files with the develop branch. As I mentioned, I have a branch that has some other needed modifications (on top of the newer g2 library) in order to read the WMO grib2 template "1" that the data is on. Again, I haven't submitted it as a PR as I'm waiting for an update to the newer stack-stack version, but you're welcome to try the branch without any promises if you just need it for testing: https://github.com/LarissaReames-NOAA/UFS_UTILS/tree/feature/chgres_rrfs_RLL

[natalie-perlin]

@natalie-perlin Are you using the develop branch here? You're not going to be able to use the full RRFS files with the develop branch. As I mentioned, I have a branch that has some other needed modifications (on top of the newer g2 library) in order to read the WMO grib2 template "1" that the data is on. Again, I haven't submitted it as a PR as I'm waiting for an update to the newer stack-stack version, but you're welcome to try the branch without any promises if you just need it for testing: https://github.com/LarissaReames-NOAA/UFS_UTILS/tree/feature/chgres_rrfs_RLL

Oh, that's great, thanks, I was not sure which branch to use!
And yes, after trying to use a updated version of g2/3.4.7, processing the files (make_ics, make_lbcs in SRW workflow tasks) still fail in the same location.

[natalie-perlin]

@LarissaReames-NOAA -
I was able to checkout your branch and build the UFS_UTILS as part of the SRW App.
With this code, the error results when running orog, i.e. in make_orog workflow task, producing
forrtl: severe (64): input conversion error, unit 5, file /proc/24463/fd/0

...
+ print_msg.sh[118]: printf '%s\n' 'Starting orography file generation...'
Starting orography file generation...
+ exregional_make_orog.sh[213]: PREP_STEP
+ exregional_make_orog.sh[0]: :
+ exregional_make_orog.sh[214]: eval time /lustre/SRW/srw-upd_ufs_utils/exec/orog
++ exregional_make_orog.sh[214]: /lustre/SRW/srw-upd_ufs_utils/exec/orog
forrtl: severe (64): input conversion error, unit 5, file /proc/24463/fd/0
Image              PC                Routine            Line        Source
orog               00000000004E03D8  Unknown               Unknown  Unknown
orog               0000000000506212  Unknown               Unknown  Unknown
orog               0000000000504C77  Unknown               Unknown  Unknown
orog               000000000040CEE2  MAIN__                     83  mtnlm7_oclsm.F
orog               000000000040CE12  Unknown               Unknown  Unknown
libc-2.17.so       00002B21C4E7E555  __libc_start_main     Unknown  Unknown
orog               000000000040CD29  Unknown               Unknown  Unknown

real    0m0.155s
user    0m0.010s
sys     0m0.016s
+ exregional_make_orog.sh[218]: print_err_msg_exit 'Call to executable (exec_fp) that generates the raw orography file returned
with nonzero exit code:
  exec_fp = "/lustre/SRW/srw-upd_ufs_utils/exec/orog"'
  ...

Are there any changes expected for orography stage?
Any suggestions are helpful!

A log file make_orog_2024060517.log.txt from the SRW workflow task that runs the orog executable is attached

make_orog_2024060517.log.txt

[LarissaReames-NOAA]

@natalie-perlin I'm honestly not sure what's going on. I have never made edits to the orog code in that branch or otherwise. It's up to date with develop other than the aforementioned changes to chgres_cube. Could you try using the develop (or whatever branch you were using previously) for everything but the make ICs/LBCs step?

[natalie-perlin]

@LarissaReames-NOAA -
If only the make_ics and make_lbcs tasks are run with the https://github.com/LarissaReames-NOAA/UFS_UTILS/tree/feature/chgres_rrfs_RLL branch, they still fail with the same error as before,

 - FATAL ERROR: READING SURFACE PRESSURE RECORD.
 - IOSTAT IS:           99
Abort(999) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 0
+ exregional_make_ics.sh[659]: print_err_msg_exit 'Call to executable (exec_fp) to generate surface and initial conditions
(ICs) files for the FV3-LAM failed:
  exec_fp = "/lustre/SRW/srw-upd_ufs_utils/exec/chgres_cube"
The external model from which the ICs files are to be generated is:
  EXTRN_MDL_NAME_ICS = "RRFS"
  ...

Could it be that I need another input files, i.e., surface files, as is mentioned in
#906 ?

Not quite sure where the surface file should be located to be used by the SRW tasks/workflow though, and at which stage (task) it is needed.