Potential problem writing parallel with compression? #264
Comments
|
Here is the piece of code: The code is writing out pfull correct, but phalf with all zeros. However the value array valueListR4 holds correct values for both pfull and phalf. |
|
What is the actual value of wrtgtid? And what is the value of n? It should be equal to wrtgrid, correct? Also, what is the ESMF_AttributGet doing? Is it getting an attribute value? |
|
Also how many processors is this running on? |
|
In your code there you only have one nf90_put_var() call writing valueListR4. You should have two calls, one to write to the phalf, and one to the pfull. You are putting the varids for both into dim_varid, but they are separate and different values. There are two vars, phalf, and pfull, and you must write to both of them. Or are you actually writing both of them in some kind of loop here? |
|
wrtGrid is an ESMF grid. Here, ESMF_AttributGet gets associated attributes
of the ESMF grid wrtGrid.
There are 40 processors running on this piece of code (I also tried 64).
The whole model code is running 948 tasks.
…On Tue, Jun 16, 2020 at 3:31 PM Edward Hartnett ***@***.***> wrote:
Also how many processors is this running on?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#264 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AI7D6TOVM7HG2XUGB4CI3S3RW7CCVANCNFSM4N74F4IQ>
.
|
|
What is the value of n please in these calls: That is, the lengths of the two dimensions. |
|
add_dim is a subroutine called to define and write the dimension for pfull
and phalf. dim_varid is a local variable defined everytime when add_dim is
called. n is the number of dimension of pfull and phalf.
.
On Tue, Jun 16, 2020 at 3:36 PM Jun Wang - NOAA Federal <jun.wang@noaa.gov>
wrote:
… wrtGrid is an ESMF grid. Here, ESMF_AttributGet gets
associated attributes of the ESMF grid wrtGrid.
There are 40 processors running on this piece of code (I also tried 64).
The whole model code is running 948 tasks.
On Tue, Jun 16, 2020 at 3:31 PM Edward Hartnett ***@***.***>
wrote:
> Also how many processors is this running on?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <#264 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AI7D6TOVM7HG2XUGB4CI3S3RW7CCVANCNFSM4N74F4IQ>
> .
>
|
|
I need the actual values of n. The lengths of the dimensions. Can you do an ncdump -h -s of the file you expect to get (i.e. the case where it works?) |
|
On Orion, I have:
netcdf atmf000 {
dimensions:
grid_xt = 3072 ;
grid_yt = 1536 ;
pfull = 127 ;
phalf = 128 ;
time = 1 ;
variables:
double grid_xt(grid_xt) ;
grid_xt:cartesian_axis = "X" ;
grid_xt:long_name = "T-cell longitude" ;
grid_xt:units = "degrees_E" ;
grid_xt:_Storage = "contiguous" ;
grid_xt:_Endianness = "little" ;
grid_xt:_NoFill = "true" ;
double lon(grid_yt, grid_xt) ;
lon:long_name = "T-cell longitude" ;
lon:units = "degrees_E" ;
lon:_Storage = "contiguous" ;
lon:_Endianness = "little" ;
lon:_NoFill = "true" ;
double grid_yt(grid_yt) ;
grid_yt:cartesian_axis = "Y" ;
grid_yt:long_name = "T-cell latiitude" ;
grid_yt:units = "degrees_N" ;
grid_yt:_Storage = "contiguous" ;
grid_yt:_Endianness = "little" ;
grid_yt:_NoFill = "true" ;
double lat(grid_yt, grid_xt) ;
lat:long_name = "T-cell latitude" ;
lat:units = "degrees_N" ;
lat:_Storage = "contiguous" ;
lat:_Endianness = "little" ;
lat:_NoFill = "true" ;
float pfull(pfull) ;
pfull:long_name = "ref full pressure level" ;
pfull:units = "mb" ;
pfull:cartesian_axis = "Z" ;
pfull:positive = "down" ;
pfull:_Storage = "contiguous" ;
pfull:_Endianness = "little" ;
pfull:_NoFill = "true" ;
float phalf(phalf) ;
phalf:_Storage = "contiguous" ;
phalf:_Endianness = "little" ;
phalf:_NoFill = "true" ;
double time(time) ;
time:long_name = "time" ;
time:units = "hours since 2019-12-14 06:00:00" ;
time:cartesian_axis = "T" ;
time:calendar_type = "JULIAN" ;
time:calendar = "JULIAN" ;
time:_Storage = "contiguous" ;
time:_Endianness = "little" ;
time:_NoFill = "true" ;
float clwmr(time, pfull, grid_yt, grid_xt) ;
clwmr:long_name = "cloud water mixing ratio" ;
clwmr:units = "kg/kg" ;
clwmr:missing_value = -1.e+10f ;
clwmr:_FillValue = -1.e+10f ;
clwmr:cell_methods = "time: point" ;
clwmr:output_file = "atm" ;
clwmr:max_abs_compression_error = 4.697358e-08f ;
clwmr:nbits = 14 ;
clwmr:_Storage = "chunked" ;
clwmr:_ChunkSizes = 1, 127, 39, 3072 ;
clwmr:_DeflateLevel = 1 ;
clwmr:_Endianness = "little" ;
clwmr:_NoFill = "true" ;
...
:source = "FV3GFS" ;
:grid = "gaussian" ;
:im = 3072 ;
:jm = 1536 ;
:_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6," ;
:_SuperblockVersion = 0 ;
:_IsNetcdf4 = 1 ;
:_Format = "netCDF-4 classic model" ;
}
On hera, I have correct phalf output as:
[Jun.Wang@hfe04 fcst.171628.hera]$ ncdump -h -s atmf000.nc |more
netcdf atmf000 {
dimensions:
grid_xt = 3072 ;
grid_yt = 1536 ;
pfull = 127 ;
phalf = 128 ;
time = 1 ;
variables:
double grid_xt(grid_xt) ;
grid_xt:cartesian_axis = "X" ;
grid_xt:long_name = "T-cell longitude" ;
grid_xt:units = "degrees_E" ;
grid_xt:_Storage = "contiguous" ;
grid_xt:_Endianness = "little" ;
grid_xt:_NoFill = "true" ;
double lon(grid_yt, grid_xt) ;
lon:long_name = "T-cell longitude" ;
lon:units = "degrees_E" ;
lon:_Storage = "contiguous" ;
lon:_Endianness = "little" ;
lon:_NoFill = "true" ;
double grid_yt(grid_yt) ;
grid_yt:cartesian_axis = "Y" ;
grid_yt:long_name = "T-cell latiitude" ;
grid_yt:units = "degrees_N" ;
grid_yt:_Storage = "contiguous" ;
grid_yt:_Endianness = "little" ;
grid_yt:_NoFill = "true" ;
double lat(grid_yt, grid_xt) ;
lat:long_name = "T-cell latitude" ;
lat:units = "degrees_N" ;
lat:_Storage = "contiguous" ;
lat:_Endianness = "little" ;
lat:_NoFill = "true" ;
float pfull(pfull) ;
pfull:long_name = "ref full pressure level" ;
pfull:units = "mb" ;
pfull:cartesian_axis = "Z" ;
pfull:positive = "down" ;
pfull:edges = "phalf" ;
pfull:_Storage = "contiguous" ;
pfull:_Endianness = "little" ;
pfull:_NoFill = "true" ;
float phalf(phalf) ;
phalf:long_name = "ref half pressure level" ;
phalf:units = "mb" ;
phalf:cartesian_axis = "Z" ;
phalf:positive = "down" ;
phalf:_Storage = "contiguous" ;
phalf:_Endianness = "little" ;
phalf:_NoFill = "true" ;
double time(time) ;
time:long_name = "time" ;
time:units = "hours since 2019-12-14 06:00:00" ;
time:cartesian_axis = "T" ;
time:calendar_type = "JULIAN" ;
time:calendar = "JULIAN" ;
time:_Storage = "contiguous" ;
time:_Endianness = "little" ;
time:_NoFill = "true" ;
float clwmr(time, pfull, grid_yt, grid_xt) ;
clwmr:long_name = "cloud water mixing ratio" ;
clwmr:_FillValue = -1.e+10f ;
clwmr:cell_methods = "time: point" ;
clwmr:output_file = "atm" ;
clwmr:max_abs_compression_error = 4.697358e-08f ;
clwmr:nbits = 14 ;
clwmr:_Storage = "chunked" ;
clwmr:_ChunkSizes = 1, 127, 39, 3072 ;
clwmr:_DeflateLevel = 1 ;
clwmr:_Endianness = "little" ;
clwmr:_NoFill = "true" ;
...
:source = "FV3GFS" ;
:grid = "gaussian" ;
:im = 3072 ;
:jm = 1536 ;
:_NCProperties =
"version=2,netcdf=4.7.4-development,hdf5=1.10.6," ;
:_SuperblockVersion = 0 ;
:_IsNetcdf4 = 1 ;
:_Format = "netCDF-4 classic model" ;
}
…On Tue, Jun 16, 2020 at 3:45 PM Edward Hartnett ***@***.***> wrote:
I need the actual values of n. The lengths of the dimensions.
Can you do an ncdump -h -s of the file you expect to get (i.e. the case
where it works?)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#264 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AI7D6TP4CVPT554JIKRSVWTRW7DWXANCNFSM4N74F4IQ>
.
|
|
OK, thanks, let me work on that for a bit... |
|
@junwang-noaa I have a draft PR up, please look and answer the questions there. Also, where is the I/O code for this? That is, what repo, what subdirectory/files, and approximately what lines of code? I would like to look at it directly... |
|
OK, this turned out to be an install issue, not a netcdf-fortran issue. NetCDF is writing parallel with compression just fine. ;-) I have added a test netcdf-fortran to confirm this, and as a starting point for the next time someone raises an issue like this. I will close this issue. |
The NOAA model is having trouble with netCDF writing some variables in parallel with compression.
Is this a netCDF problem? That is the question. I will add a test to mimic the situation and we will see if we can catch a bug, or confirm that netCDF is doing what it should.
The text was updated successfully, but these errors were encountered: