Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update sfc_climo_gen to output fractional vegetation and soil type #709

Closed
GeorgeGayno-NOAA opened this issue Oct 28, 2022 · 19 comments · Fixed by #748
Closed

Update sfc_climo_gen to output fractional vegetation and soil type #709

GeorgeGayno-NOAA opened this issue Oct 28, 2022 · 19 comments · Fixed by #748
Assignees
Labels
enhancement New feature or request

Comments

@GeorgeGayno-NOAA
Copy link
Collaborator

Currently, this program outputs a dominate category at each model point. Update the program to output the percentage of each category within a model point.

@GeorgeGayno-NOAA GeorgeGayno-NOAA added the enhancement New feature or request label Oct 28, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 28, 2022
Routines were created to interpolate, search for missing data, and
output the data.

Fixes ufs-community#709.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 28, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 28, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 31, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 31, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 31, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 31, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 31, 2022
Add the new logic for the processing of soil type.

Fixes ufs-community#709.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 1, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 2, 2022
@GeorgeGayno-NOAA
Copy link
Collaborator Author

Tried some timing and memory tests on Hera using 4adcb64. I used a small (301x200) CONUS ESG grid and the VIIRS vegetation data.

  • Test 1 - use the vegetation_type.viirs.igbp.conus.30s.nc as input. Using one node and 24 tasks, it ran in about 60 seconds.
  • Test 2 - use the vegetation_type.viirs.igbp.nh.30s.nc as input. I had to request the two 'bigmem' nodes, 12 tasks per node. It ran in about six minutes.

So, the fractional option can take a lot of memory. We are pushing the limit using the NH version of the file. Using the global version may not be feasible on some machines.

@GeorgeGayno-NOAA
Copy link
Collaborator Author

Tried some timing and memory tests on Hera using 4adcb64. I used a small (301x200) CONUS ESG grid and the VIIRS vegetation data.

  • Test 1 - use the vegetation_type.viirs.igbp.conus.30s.nc as input. Using one node and 24 tasks, it ran in about 60 seconds.
  • Test 2 - use the vegetation_type.viirs.igbp.nh.30s.nc as input. I had to request the two 'bigmem' nodes, 12 tasks per node. It ran in about six minutes.

So, the fractional option can take a lot of memory. We are pushing the limit using the NH version of the file. Using the global version may not be feasible on some machines.

I repeated this test on WCOSS2 using the NH file. It was able to run on one node/24 tasks, but I had to request 500 GB of memory: #PBS -l select=1:ncpus=24:mem=500GB. It took about 6 minutes to run.

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 14, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 14, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 15, 2022
@GeorgeGayno-NOAA
Copy link
Collaborator Author

When processing a fractional grid (points can be a mix of land and non-land), how will the dominate category be defined? Suppose a point is:

  • 40% land
  • 60% non-land
  • 35% forest
  • 5% grassland

Is the dominate category forest or water?

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 15, 2022
@GeorgeGayno-NOAA
Copy link
Collaborator Author

In order to read the water flag from the file, the file attributes and missing values need to be changed. This may be done using netCDF tools. On WCOSS2, the vegetation type files were changed as follows:

module load gsl
module load nco

ncatted -h -a water_category,vegetation_type,c,i,17 file.nc
ncatted -h -a missing_value,vegetation_type,o,i,17 file.nc
ncap2 -O -s 'where(vegetation_type < 0) vegetation_type=17' file.nc file.nc

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 16, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 16, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 17, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 21, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 21, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 21, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 21, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 29, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 29, 2022
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 6, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 9, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 9, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 9, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 9, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 9, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 10, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 19, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 25, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 30, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Feb 2, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Feb 2, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Feb 6, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Feb 6, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Feb 9, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Feb 16, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Feb 16, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Feb 22, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Feb 24, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Feb 28, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Mar 2, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Mar 8, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Mar 17, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Mar 20, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Mar 27, 2023
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants