Writing using hyperslabs is extremely slow compared to netCDF-C

[abhibaruah]

netCDF4 version = 1.6.1
netCDF4 hdf5libversion = 1.12.1
netCDF4 netcdf4libversion = 4.8.1

OS: Windows 10

I am trying to create a netCDF4 file and write to it using hyperslabs. However, the writing is extremely slow. The repro code that I posted below takes over 100 seconds. The same workflow using netCDF-C takes a maximum of 10 seconds.

In the repro code below, if I change the 'stride' and 'count' to be (1,1,1) and (400,400,400) respectively, the repro takes < 10 sec.

Is this expected with hyperslab writing or some kind of performance bug?

# Import required packages
from netCDF4 import Dataset
from time import perf_counter
import numpy as np
  
# Define file name and variable parameters
filename = 'tempfile_python.nc'
varName = 'xlrg_var'

# Variable parameters
start = (0,0,0)
stride = (2,2,2)
count = (200,200,200)
varSize = (400,400,400)

inpData = np.round(np.random.rand(count[0],count[1],count[2]),3)
      
# Start counter
tInit = perf_counter()
  
# Create file and write data to the variable
rootgrp = Dataset(filename,"w",format="NETCDF4")

  
# Create variable and add data using hyperslab parameters
dimObj_400 = rootgrp.createDimension("dim_400",varSize[2])
inpDims = ("dim_400","dim_400","dim_400")
varObj = rootgrp.createVariable(varName,"f8",inpDims)
(varObj[start[0]:varSize[0]:stride[0],start[1]:varSize[1]:stride[1],
        start[2]:varSize[2]:stride[2]]) = inpData
    
# Close file
rootgrp.close()
  
# Stop counter and return recorded time
execTime = perf_counter() - tInit
print("===Time taken:%f===" % (execTime))

[jswhit]

Can we see your C code? The python interface uses nc_put_vara if the strides are all 1, and nc_put_vars for strided writes. The slowness of nc_put_vars is a known issue, which was substantially improved by this update to the C lib which was merged in 2018. What version of the C lib are you using? (you can determine this by printing netCDF4.__netcdf4libversion__)).

[jswhit]

I see what is going on - we had added a workaround for the slowness of nc_put_vars that splits the write into separate un-strided writes using nc_put_vara. Now that nc_put_vars has been sped up, this workaround is no longer needed. Intial tests show that using nc_put_vars cuts the run time by two orders of magnitude. PR is forthcoming.

[abhibaruah]

Thanks @jswhit for taking a look.
Do you still need my C repro? Actually, I was comparing the performance in Python to MATLAB (which takes a max of 10 sec, so I am guessing that pure C code would be even faster), which uses the netCDF-C library underneath.
For MATLAB, the netCDF-C library version is v4.8.1.

If it is needed, I can convert my MATLAB code to C repro.

[jswhit]

No need for the C code @abhibaruah. PR #1224 should solve it - if you can give it a try and verify that it does that would be great.

[abhibaruah]

Tested it on my repro. The timing came down from 100+ seconds to ~8 seconds.
Thanks for fixing this.

[jswhit]

closed by PR #1124