OSError: NetCDF: HDF error if netCDF4 and h5py installed together

[eamag]

from netCDF4 import Dataset
import os

data = Dataset(os.path.join(filepath, file), 'r', format='NETCDF4')

File "netCDF4/_netCDF4.pyx", line 1848, in netCDF4._netCDF4.Dataset.init (netCDF4/_netCDF4.c:13983)
OSError: NetCDF: HDF

netCDF4==1.2.7
h5py==2.7.0

[matthew-brett]

How did you install netCDF4, h5py? What system are you on? Which Python are you using?

[matthew-brett]

@eamag - unfortunately, if we don't have any more information from you about your system, it's going to be hard for us to reproduce your problem and fix it.

[eamag]

Sorry for late response, @matthew-brett
All packages were installed via pip3, ubuntu 16.04.2, python 3.5.2

[matthew-brett]

No problem about the late response. Here's what I did to try and replicate. I started a fresh Ubuntu 16.04 container:

docker run --rm -ti ubuntu:16.04 /bin/bash

Then I ran these commands in the container, which didn't give me an error:

apt-get update
apt-get install -y wget python3.5 
wget https://bootstrap.pypa.io/get-pip.py
python3.5 get-pip.py --user
~/.local/bin/pip3 install h5py
~/.local/bin/pip3 install netCDF4
wget https://www.unidata.ucar.edu/software/netcdf/examples/test_hgroups.nc
python3.5 -c "import netCDF4; netCDF4.Dataset('test_hgroups.nc', 'r', format='NETCDF4')"

Is there something special about your file that would generate the error? Can you put it somewhere where I can get it to test with?

[matthew-brett]

Also - just to check - could you try uninstalling and installing again:

pip3 uninstall -y h5py netcdf4
pip3 install --user h5py netcdf4

[jennirinker]

I am getting this issue and can probably help with recreating it.

MWE:

import netCDF4 as nc4
f = nc4.Dataset('sample.nc','w', format='NETCDF4')

Packages:

• NetCDF4 1.4.0 installed using conda (build py36hfa18eed_1)
• h5py 2.7.1 installed using pip

I assume this is pip and conda not playing nicely together. I'm going to mess around a bit and see if I can get a workaround.

Pinging @matthew-brett

[jennirinker]

Solved it: I just uninstalled the version of h5py that was installed with pip.

Interestingly, some other package I have must include h5py as a hidden requirement because it's not shown in pip list or conda list but import h5py works just fine. Magic.

[randallpittman]

Workaround: If netCDF4 is imported first, there is no error and everything works fine!

Any relation to #694?

I need to use h5py and netcdf4-python together. My setup:
OS: Fedora 31
Python: pyenv 3.8.1 virtualenv
netCDF4: 1.5.3
h5py: 2.10.0

Code to cause the problem:

import h5py
import netCDF4

ds = netCDF4.Dataset("my_dataset.nc")

Output:

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-6-be938cfa3e55> in <module>
----> 1 ds = netCDF4.Dataset("/data/new/NewportTower_20200227_001106.nc")

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.__init__()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

OSError: [Errno -101] NetCDF: HDF error: b'/data/new/NewportTower_20200227_001106.nc'

[arthur-e]

I can confirm the order in which h5py and netCDF4 are imported makes a difference. Importing netCDF4 first will fix this error.

[randallpittman]

I'd like to just say (with all the love in my heart) that I sure do wish netCDF4-python would be a little less opaque about its errors.

[jswhit]

I'm open to suggestions on how to better interpret errors coming from the HDF5 c library.

[randallpittman]

I'm open to suggestions on how to better interpret errors coming from the HDF5 c library.

Fair enough--I guess it's more of an upstream problem.

[randallpittman]

@jswhit

I'd like to just say (with all the love in my heart) that I sure do wish netCDF4-python would be a little less opaque about its errors.

I'm open to suggestions on how to better interpret errors coming from the HDF5 c library.

This one's a bad example of what I was thinking of. I more meant stuff like "not a valid ID" when really it could say, "You closed the dataset, dummy, so you can't access its contents anymore."

I've been working on (and using in my projects) a subclass of Dataset that does some nice things like keep the dataset's opening arguments stateful, allow it to be reopened, and give tell you nicely if you're trying to work with a closed dataset. https://gitlab.com/osu-nrsg/ncdataset-python/-/blob/master/ncdataset/_ncdataset.py

[LSRathore]

I am having the same issue while working on the HPC cluster, however it works fine on the local computer.

My setup-
python 3.6.5
netCDF4 1.5.3
h5py 2.10.0

import netCDF4 as nc
import h5py
data=nc.Dataset("ncfile.nc")

Any solutions to this problem?

[hydronow]

I am having the same issue while working on the HPC cluster, however it works fine on the local computer.

My setup-
python 3.6.5
netCDF4 1.5.3
h5py 2.10.0

import netCDF4 as nc
import h5py
data=nc.Dataset("ncfile.nc")

Any solutions to this problem?

I am having the same issue with data from the ESGF website (https://esgf-node.llnl.gov/search/esgf-llnl/) .

[LSRathore]

@hydronow I was getting the error because my netcdf file was incomplete when I transferred it to the cluster. My issue got resolved when I transferred the file again.

[hydronow]

@lokendrarathore I think you right. At this point I am somewhat confident that the file itself is broken and that the netCDF4 library is working fine.