Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert the high-resolution mask and terrain data to NetCDF #900

Merged
merged 27 commits into from
Mar 7, 2024
Merged

Convert the high-resolution mask and terrain data to NetCDF #900

merged 27 commits into from
Mar 7, 2024

Conversation

GeorgeGayno-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA GeorgeGayno-NOAA commented Feb 22, 2024 •
edited
Loading

DESCRIPTION OF CHANGES:

  • Baseline utilities that convert to NetCDF the mask and terrain data used as input to the orography code. Currently, a simple binary format is used. The utilities were designed to be used only once. Therefore, their build is turned off by default.
  • Update the orography source code to read the converted files.
  • Remove some unused logic and variables.

TESTS CONDUCTED:

  • Compile branch on all Tier 1 machines using Intel (Hera, Orion, Jet, Hercules and WCOSS2). Done using 3fae81f.
  • Compile branch on Hera using GNU. Done use 3fae81f.
  • Compile branch in 'Debug' mode on WCOSS2. Done using 3fae81f on Cactus.
  • Run unit tests locally on any Tier 1 machine. Done on Cactus using 3fae81f.
  • Run relevant consistency tests locally on all Tier 1 machine. On Hera/Hercules/Orion/Jet/WCOSS2, all grid_gen consistency tests passed using 3fae81f.

Describe any additional tests performed.

  • Doxygen was created with no warnings on Hera using 3fae81f.

DEPENDENCIES:

After the merge, the new files will need to be placed in the official directories on all supported machines and on AWS (https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#fix/fix_orog/)

The AWS contacts are: sylvia.chin@noaa.gov and edward.snyder@noaa.gov

DOCUMENTATION:

Updated: https://georgegayno-noaaufs-utils.readthedocs.io/en/orog_nc/ufs_utils.html#id30

ISSUE:

Fixes #786.

CONTRIBUTORS:

@barlage

@GeorgeGayno-NOAA
Add small program to read UMD mask file, used by the orog
5a05e1b 
code, and convert it to netcdf.

Fixes ufs-community#786.
@GeorgeGayno-NOAA
Update program that converts umd mask data to netcdf.
d9f17f3 
Fixes ufs-community#786.
@GeorgeGayno-NOAA
Merge branch 'develop' into orog_nc
e1be717 
Fixes ufs-community#786.
@GeorgeGayno-NOAA
Update to read the UMD mask data in netcdf format. Remove
fac421b 
old mask read options.

Fixes ufs-community#786.
@GeorgeGayno-NOAA
Baseline program to convert the gmted2010 data to netcdf.
3eef445 
Fixes ufs-community#786.
@GeorgeGayno-NOAA
Update orog code to read gmted2010 data in netcdf.
d91fa8e 
Fixes ufs-community#786.
@GeorgeGayno-NOAA
Add program to convert Antarctica RAMP data to netcdf.
ed943ca 
Fixes ufs-community#786.
@GeorgeGayno-NOAA
Updates to ramp data conversion utility.
338c506 
Fixes ufs-community#786.
@GeorgeGayno-NOAA
Merge branch 'develop' into orog_nc
c67400e 
Fixes ufs-community#786.
@GeorgeGayno-NOAA
Add read of Antarctic RAMP data.
ce3b39a 
Fixes ufs-community#786.
@GeorgeGayno-NOAA
Remove 'stop' processing.
d600b50 
Fixes ufs-community#786.
@GeorgeGayno-NOAA
Cleanup the read of the RAMP data.
cb49326 
Fixes ufs-community#786.
@GeorgeGayno-NOAA
Remove some unused variables from orog code.
d8ba87c 
Fixes ufs-community#786.
@GeorgeGayno-NOAA
Update mask conversion program to add unit attributes.
dda6411 
Fixes ufs-community#786.
@GeorgeGayno-NOAA GeorgeGayno-NOAA self-assigned this Feb 22, 2024
@GeorgeGayno-NOAA
Copy link
Collaborator Author

GeorgeGayno-NOAA commented Feb 22, 2024 •
edited
Loading

The converted datasets are on Hera: /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS/fix/orog_raw.test

  • landcover.umd.30s.nc
  • topography.gmted2010.30s.nc
  • topography.antarctica.ramp.30s.nc

and will need to be placed in the official 'fixed' locations on all machines. And on AWS:

@barlage
Copy link
Collaborator

barlage commented Feb 22, 2024

@GeorgeGayno-NOAA were you able to get reproducibility using the netcdf versions? When I was playing with this a while back, I was able to with land and topo, but the Antarctic did not.

@GeorgeGayno-NOAA
Copy link
Collaborator Author

@GeorgeGayno-NOAA were you able to get reproducibility using the netcdf versions? When I was playing with this a while back, I was able to with land and topo, but the Antarctic did not.

Yes. All the grid_gen regression tests passed on Hera.

@GeorgeGayno-NOAA GeorgeGayno-NOAA marked this pull request as ready for review February 28, 2024 16:24
barlage
barlage suggested changes Mar 4, 2024
View reviewed changes
CMakeLists.txt Outdated Show resolved Hide resolved
sorc/orog_mask_tools.fd/orog.fd/mtnlm7_oclsm.F Outdated Show resolved Hide resolved
sorc/orog_mask_tools.fd/orog.fd/mtnlm7_oclsm.F Outdated Show resolved Hide resolved
CMakeLists.txt Outdated Show resolved Hide resolved
sorc/orog_mask_tools.fd/orog_netcdf.fd/mask.fd/mask.f90 Outdated Show resolved Hide resolved
sorc/orog_mask_tools.fd/orog_netcdf.fd/ramp.fd/ramp.f90 Outdated Show resolved Hide resolved
sorc/orog_mask_tools.fd/orog_netcdf.fd/topo.fd/topo.f90 Outdated Show resolved Hide resolved
@GeorgeGayno-NOAA
Copy link
Collaborator Author

The converted datasets are on Hera: /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS/fix/orog_raw.test

  • landcover.umd.30s.nc
  • topography.gmted2010.30s.nc
  • topography.antarctica.ramp.30s.nc

and will need to be placed in the official 'fixed' locations on all machines. And on AWS:

@barlage - Are the file records descriptive enough? (i.e., do an ncdump -h on them).

@barlage
Copy link
Collaborator

barlage commented Mar 5, 2024

@GeorgeGayno-NOAA I took a look at those yesterday on Hera and they look good to me. Thanks for changing the names.

@GeorgeGayno-NOAA
Copy link
Collaborator Author

Will rerun all tests using 3fae81f.

@GeorgeGayno-NOAA GeorgeGayno-NOAA mentioned this pull request Mar 6, 2024
Closed
4 tasks
@GeorgeGayno-NOAA
Copy link
Collaborator Author

As of today, the files are on the cloud: https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#fix/fix_orog/

And they were place on all supported machines: NOAA-EMC/global-workflow#2378

Will merge.

@GeorgeGayno-NOAA GeorgeGayno-NOAA merged commit 658c351 into ufs-community:develop Mar 7, 2024
4 checks passed
@GeorgeGayno-NOAA GeorgeGayno-NOAA deleted the orog_nc branch April 12, 2024 17:22
BinLiu-NOAA pushed a commit to hafs-community/UFS_UTILS that referenced this pull request Jul 10, 2024
@GeorgeGayno-NOAA @BinLiu-NOAA
Update orog code to read high-resolution mask and terrain data in Net…
7df798c 
…CDF (ufs-community#900)

Update orog program to read NetCDF versions of the UMD mask, 
GMTED2010 terrain and the RAMP terrain. Also, remove some unused
variables and logic.

Baseline the programs that were used to convert these datasets from
simple binary to NetCDF.
 
Fixes ufs-community#786.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@barlage barlage barlage approved these changes

Assignees

@GeorgeGayno-NOAA GeorgeGayno-NOAA

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Change orog_mask_tools inputs from binary to netcdf
2 participants
@GeorgeGayno-NOAA @barlage