testing parallel writes just as NOAA does them

[edwardhartnett]

This is a draft PR, put up so some NOAA collaborators can inspect the code and we can make this test mirror a problem area of NOAA code.

This is part of #264

@junwang-noaa please inspect the code in f90tst_paralllel_compressed.F90. This produces a file with this ncdump:

netcdf f90tst_parallel_compressed {
dimensions:
	grid_xt = 3072 ;
	grid_xy = 1536 ;
	pfull = 127 ;
	phalf = 128 ;
	time = 1 ;
variables:
	double grid_xt(grid_xt) ;
		grid_xt:_Storage = "contiguous" ;
		grid_xt:_Endianness = "little" ;
	double lon(grid_xy, grid_xt) ;
		lon:_Storage = "contiguous" ;
		lon:_Endianness = "little" ;
	double grid_yt(grid_xy) ;
		grid_yt:_Storage = "contiguous" ;
		grid_yt:_Endianness = "little" ;
	double lat(grid_xy, grid_xt) ;
		lat:_Storage = "contiguous" ;
		lat:_Endianness = "little" ;
	float pfull(pfull) ;
		pfull:_Storage = "contiguous" ;
		pfull:_Endianness = "little" ;
	float phalf(phalf) ;
		phalf:_Storage = "contiguous" ;
		phalf:_Endianness = "little" ;
	double time(time) ;
		time:_Storage = "contiguous" ;
		time:_Endianness = "little" ;
	float clwmr(time, pfull, grid_xy, grid_xt) ;
		clwmr:_Storage = "contiguous" ;
		clwmr:_Endianness = "little" ;

// global attributes:
		:_NCProperties = "version=2,netcdf=4.8.0-development,hdf5=1.10.6," ;
		:_SuperblockVersion = 0 ;
		:_IsNetcdf4 = 1 ;
		:_Format = "netCDF-4" ;
}

[edwardhartnett]

@junwang-noaa some questions:

1 - Do you define all dims first, then all vars? Or do you intermix? What is the order of dim and var creation, exactly?

2 - Also, how is the data written? There are 40 processors involved, how do they split writing of the phalf field? Do they each write 1/40th? Or does one processor write it all?

3 - You do an enddef/redef for each def_var, right? What is the exact ordering of calls there?

[junwang-noaa]

@junwang-noaa some questions:

Do you define all dims first, then all vars? Or do you intermix? What is the order of dim and var creation, exactly?

Also, how is the data written? There are 40 processors involved, how do they split writing of the phalf field? Do they each write 1/40th? Or does one processor write it all?

We do intermix. the processors does not split writing of the phalf. Please see the code below, the valueListR4 is a array with full 128 elements of phalf.

nf90_create(trim(filename),&
cmode=IOR(IOR(NF90_CLOBBER,NF90_NETCDF4),NF90_CLASSIC_MODEL),&
comm=mpi_comm, info = MPI_INFO_NULL, ncid=ncid)
nf90_set_fill(ncid, NF90_NOFILL, oldMode);
nf90_def_dim(grid_xt...)
nf90_def_dim(grid_yt...)
nf90_def_var(...grid_xt...)
nf90_def_var(...lon...)
nf90_def_var(...grid_yt...)
nf90_def_var(...lat..._)
call add_dim("pfull"...)
call add_dim("phalf"...)
call add_dim("time"...)

subroutine add_dim(dim_name...)
nf90_def_dim(dim_name...)
nf90_def_var(..dim_name..dim_varid..)
nf90_enddef
nf90_put_var (ncid, dim_varid, values=valueListR4)
nf90_redef
end subroutine add_dim

[edwardhartnett]

Where is the code that writes the values to the lat/lon variables?

[edwardhartnett]

OK, I believe this code is pretty close to what NOAA code does. I had to come up with my own (checkerboard) decomposition.

[WardF]

Thanks!

[edwardhartnett]

@WardF thanks for the quick merge here.

I will have a C version of this program (even more like the NOAA code, and with built-in timing) soon. Then this test will migrate to PIO and there become a PIO performance benchmark.

So we see how a simple test program can be very useful in many different projects. ;-)