Update sfc_climo_gen to include soil color dataset needed by noahmp land model

[barlage]

Noah-MP land model defines soil albedo through a lookup table based on a soil color classification. There is a readily available dataset developed for the CLM/CTSM land model (see Lawrence and Chase(2007) JGR, doi:10.1029/2006JG000168) that can be used. This dataset has already been formatted to be consistent with sfc_climo_gen input requirements (/scratch2/NCEPDEV/land/data/input_data/soil_color/soil_color.clm.0.05.nc). The capability to produce this data on the UFS grid is needed.

[GeorgeGayno-NOAA]

@KateFriedman-NOAA Can you please add this file to the ./fix/sfc_climo directory on each machine - /scratch2/NCEPDEV/land/data/input_data/soil_color/soil_color.clm.0.05.nc

No hurry. Thanks!

[RongqianYang-NOAA]

The tiled soil color data on various fractional grids (C48,C96,C192,C384,C768 and C1152) is
available on Hera under "/scratch2/NCEPDEV/land/data/input_data/soil_color_clm_tiled/".
The data was generated using the 'sfc_climo_gen' utility of UFS_utils , with code changes made
necessary for the addition of the new soil color field. The addition of the soil color field follows
exactly the same way as for the soil type field. The existing fractional orography data under
"/scratch1/NCEPDEV/global/glopara/fix/orog/20220805/'RES'.'OCN'_frac/" was used in the
generation runs (where 'RES' represents resolution - e.g.,'C48','C96' ..., and 'OCN' is the
resolution of MOM6 used in the water fraction creation - e.g.,'mx500' for C48, 'mx100' for C96 ...).

[KateFriedman-NOAA]

@KateFriedman-NOAA Can you please add this file to the ./fix/sfc_climo directory on each machine - /scratch2/NCEPDEV/land/data/input_data/soil_color/soil_color.clm.0.05.nc

I've pulled that file into the fix set on Hera an rsync'd it to Jet, Orion, and WCOSS2-Dogwood. Will be available on WCOSS2-Cactus as soon as the dev_transfer rsync job starts.

[KateFriedman-NOAA]

I should also have pointed out, it's in the newer 20221017 sfc_climo subfolder:

[glopara@hfe12 20221017]$ pwd
/scratch1/NCEPDEV/global/glopara/fix/sfc_climo/20221017
[glopara@hfe12 20221017]$ ll soil_color.clm.0.05.nc 
-rw-r--r-- 1 glopara global 103865024 Dec  8 19:53 soil_color.clm.0.05.nc

[GeorgeGayno-NOAA]

I should also have pointed out, it's in the newer 20221017 sfc_climo subfolder:

[glopara@hfe12 20221017]$ pwd
/scratch1/NCEPDEV/global/glopara/fix/sfc_climo/20221017
[glopara@hfe12 20221017]$ ll soil_color.clm.0.05.nc 
-rw-r--r-- 1 glopara global 103865024 Dec  8 19:53 soil_color.clm.0.05.nc

Thanks, @KateFriedman-NOAA, I did not realize that file was already hosted.

@RongqianYang-NOAA has mapped this data to our standard fractional grids - ./orog/C???.mx???_frac. At some point, he will want his data added to those directories.

[KateFriedman-NOAA]

Thanks, @KateFriedman-NOAA, I did not realize that file was already hosted.

In case my words confused...it wasn't already in there, I added it new to that specific timestamp subfolder. :)

[StevePny]

@GeorgeGayno-NOAA @KateFriedman-NOAA

The file soil_color.clm.0.05.nc is not available in the public link pointed to by:
https://noaa-emcufs-utils.readthedocs.io/en/latest/ufs_utils.html

Which points to:
https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#fix/fix_sfc_climo/

I also do not see it at any of these locations:
https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix/
https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix_nco_gfsv16/
https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix_nco_gfsv15/

This missing file is currently breaking my execution of fv3gfs_driver_grid.sh

 - OPEN FILE: /UFS_UTILS/ush/..//fix/sfc_climo/soil_color.clm.0.05.nc
 
 FATAL ERROR: OPENING INPUT SOURCE FILE: No such file or directory

Is there a place this file can be found online?

[GeorgeGayno-NOAA]

@GeorgeGayno-NOAA @KateFriedman-NOAA

The file soil_color.clm.0.05.nc is not available in the public link pointed to by: https://noaa-emcufs-utils.readthedocs.io/en/latest/ufs_utils.html

Which points to: https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#fix/fix_sfc_climo/

I also do not see it at any of these locations: https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix/ https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix_nco_gfsv16/ https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix_nco_gfsv15/

This missing file is currently breaking my execution of fv3gfs_driver_grid.sh

 - OPEN FILE: /UFS_UTILS/ush/..//fix/sfc_climo/soil_color.clm.0.05.nc
 
 FATAL ERROR: OPENING INPUT SOURCE FILE: No such file or directory

Is there a place this file can be found online?

@JeffBeck-NOAA Do you know who hosts the 'fixed' data to amazonaws?

[GeorgeGayno-NOAA]

@GeorgeGayno-NOAA @KateFriedman-NOAA

The file soil_color.clm.0.05.nc is not available in the public link pointed to by: https://noaa-emcufs-utils.readthedocs.io/en/latest/ufs_utils.html

Which points to: https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#fix/fix_sfc_climo/

I also do not see it at any of these locations: https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix/ https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix_nco_gfsv16/ https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix_nco_gfsv15/

This missing file is currently breaking my execution of fv3gfs_driver_grid.sh

 - OPEN FILE: /UFS_UTILS/ush/..//fix/sfc_climo/soil_color.clm.0.05.nc
 
 FATAL ERROR: OPENING INPUT SOURCE FILE: No such file or directory

Is there a place this file can be found online?

@StevePny You don't have access to our NOAA RDHPCS machines?

[StevePny]

@GeorgeGayno-NOAA No longer, I'm running everything on AWS ec2 right now. Also, in general for the UFS community many others will not have access to RDHPCS.

[GeorgeGayno-NOAA]

@GeorgeGayno-NOAA @KateFriedman-NOAA

The file soil_color.clm.0.05.nc is not available in the public link pointed to by: https://noaa-emcufs-utils.readthedocs.io/en/latest/ufs_utils.html

Which points to: https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#fix/fix_sfc_climo/

I also do not see it at any of these locations: https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix/ https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix_nco_gfsv16/ https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix_nco_gfsv15/

This missing file is currently breaking my execution of fv3gfs_driver_grid.sh

 - OPEN FILE: /UFS_UTILS/ush/..//fix/sfc_climo/soil_color.clm.0.05.nc
 
 FATAL ERROR: OPENING INPUT SOURCE FILE: No such file or directory

Is there a place this file can be found online?

@KateFriedman-NOAA do you host the files on https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix/fix_sfc_climo/

[KateFriedman-NOAA]

@KateFriedman-NOAA do you host the files on https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/GFS/fix/fix_sfc_climo/

I do. Apologies, I see we failed to keep the ftp server fix set updated with the on-disk sets. Let me look into doing this, I need to make sure we have the room on the ftp server since we've tripled the fix set since it was last updated there. Stay tuned...

[KateFriedman-NOAA]

Note the restructuring that happened last year would also occur in the fix set if I update it with the current state of things.

[KateFriedman-NOAA]

@JeffBeck-NOAA Do you know who hosts the 'fixed' data to amazonaws?

I second @GeorgeGayno-NOAA 's question. Sounds like that set needs to get updated. I'm not getting good news on available space on ftp to keep adding fix files there.

[JeffBeck-NOAA]

@MichaelLueken, could you put us in contact with whomever manages the fixed files on AWS? Thanks!

[MichaelLueken]

@JeffBeck-NOAA, I'm working on trying to figure out to put you in contact with to add this fix file to AWS. How urgent is it that the file is added? Thanks!

[GeorgeGayno-NOAA]

@JeffBeck-NOAA, I'm working on trying to figure out to put you in contact with to add this fix file to AWS. How urgent is it that the file is added? Thanks!

@MichaelLueken The file was requested by @StevePny.

[MichaelLueken]

@StevePny I have been asked to see the urgency of making this file available on AWS. Please let me know if you still require this file so that I can let the Project Owner of the team who would need to add this file know. Thanks!

[StevePny]

Hi @MichaelLueken, at the moment, the absence of this file is crashing my use of UFS_UTILS to generate grids, as of the change associated with this issue (~December 8). This breaks the functionality described at https://noaa-emcufs-utils.readthedocs.io/en/latest/ufs_utils.html for any outside users. I suppose I can pull a tag from before this change in the meantime, but in general I would consider that high priority.

For example, this documentation says:

The surface climatological data is located here [./fix/fix_sfc_climo](https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#fix/fix_sfc_climo/). All NetCDF.

Global 1-degree fractional coverage strong/weak zenith angle albedo - facsf.1.0.nc
Global 0.05-degree maximum snow albedo - maximum_snow_albedo.0.05.nc
Global 2.6 x 1.5-degree soil substrate temperature - substrate_temperature.2.6x1.5.nc
Global 0.05-degree four component monthly snow-free albedo - snowfree_albedo.4comp.0.05.nc
Global 1.0-degree categorical slope type - slope_type.1.0.nc
Global 0.05-degree CLM soil color (Lawrence and Chase, 2007 JGR) - soil_color.clm.0.05.nc

and yet the file soil_color.clm.0.05.nc is not in the location stated.

[JeffBeck-NOAA]

@StevePny, the soil color file now exists in the AWS bucket. Can you check that it works for you? Thanks!

[StevePny]

Hi @JeffBeck-NOAA, thanks yes I am able to wget the file now and run both ./fv3gfs_driver_grid.sh and ./chgres_cube.sh successfully.

[JeffBeck-NOAA]

Thanks @StevePny, glad to hear it's working! @MichaelLueken, thanks for your help!