Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make netcdf-4 use the the stride > 1 facilities of hdf5 #908

Closed
DennisHeimbigner opened this issue Mar 21, 2018 · 4 comments
Closed

Make netcdf-4 use the the stride > 1 facilities of hdf5 #908

DennisHeimbigner opened this issue Mar 21, 2018 · 4 comments
Labels
area/netcdf4 type/enhancement type/performance performance related issues

Comments

@DennisHeimbigner
Copy link
Collaborator

Currently, netcdf-4 uses a very slow implementation of vars when stride > 1.
Convert to use the stride capabilities of the underlying hdf5.
@jswhit jswhit mentioned this issue Mar 21, 2018
Closed
@czender
Copy link
Contributor

czender commented Mar 27, 2018

FYI NCO long ago implemented a workaround that addresses this issue for the special case when the strided dimension is the first dimension and only one dimension is strided. The code flag is USE_NC4_SRD_WORKAROUND in nco_msa.c. Once the library fix for #908 is done then we can deprecate our workaround. Pinging @hmb1 so he is aware of this pending development.

@jswhit jswhit mentioned this issue May 13, 2018
Open
@DennisHeimbigner
Copy link
Collaborator Author

Thanks for the reminder. I am starting work on this just now.

@DennisHeimbigner
Copy link
Collaborator Author

Ok, I think I have this working. It looks like it is going to make
a very big performance improvement. For a simple test
on a 16000000 integer variable with stride 2X2, the difference
I see is a factor of about 150.
Your mileage may vary.
Anyway, as soon as it passes all the relevant tests, I will turn it into
a pull request.

DennisHeimbigner added a commit that referenced this issue May 22, 2018
@DennisHeimbigner
Re-Implement the nc_get/put_vars operations for netcdf-4 using the
ee509ff 
corresponding HDF5 operations.

re: github issue #908
also in reference to pydata/xarray#2004

The netcdf-c library has implemented the nc_get_vars and nc_put_vars
operations as element at a time. This has resulted in very slow
operation.

This pr attempts to improve the situation for netcdf-4/hdf5 files
by using the slab operations provided by the hdf5 library. The new
implementation passes the get/put vars stride information down to
the hdf5 slab operations.

The result appears to improve performance significantly. Some simple
tests on large 2-D arrays shows speedups in excess of 150.

Misc. other changes:
1. fix bug in ncgen/semantics.c; using a list's allocated length
   instead of actual length.
2. Added a temporary hook in the netcdf library plus a performance
   test case (tst_varsperf.c) to estimate the speedup. After users
   have had some experience with this, I will remove it, probably
   after the 4.7 release.
@DennisHeimbigner DennisHeimbigner mentioned this issue May 22, 2018
Merged
@DennisHeimbigner
Copy link
Collaborator Author

This is now solved by pull request #1001

@jswhit jswhit mentioned this issue Dec 6, 2022
Closed
@abhibaruah abhibaruah mentioned this issue Jul 19, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/netcdf4 type/enhancement type/performance performance related issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@DennisHeimbigner @czender