Getting fill values when reading from external dataset

[ajelenak]

External dataset's data are in a raw binary file external to its HDF5 file. In some cases reading from an external dataset contains more zeroes (same as the dataset's fill value) than when directly reading from the binary file. The HDF5/binary files combo seems to represent LOFAR data.

The external dataset shape is (228864, 6400) and datatype is float32. The binary file is 5,858,918,400 bytes.

This problem was reported as h5py/h5py#2394.

Based on one file's attribute, it seems the HDF5 file was created with the 1.8.12 version. The user is using the 1.12.2 version. A Python script I created to test this case using the 1.14.3 version fails with this error:

HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0:
  #000: /Users/ajelenak/Documents/h5py/hdf5/src/H5D.c line 1061 in H5Dread(): can't synchronously read data
    major: Dataset
    minor: Read failed
  #001: /Users/ajelenak/Documents/h5py/hdf5/src/H5D.c line 1008 in H5D__read_api_common(): can't read data
    major: Dataset
    minor: Read failed
  #002: /Users/ajelenak/Documents/h5py/hdf5/src/H5VLcallback.c line 2092 in H5VL_dataset_read_direct(): dataset read failed
    major: Virtual Object Layer
    minor: Read failed
  #003: /Users/ajelenak/Documents/h5py/hdf5/src/H5VLcallback.c line 2048 in H5VL__dataset_read(): dataset read failed
    major: Virtual Object Layer
    minor: Read failed
  #004: /Users/ajelenak/Documents/h5py/hdf5/src/H5VLnative_dataset.c line 373 in H5VL__native_dataset_read(): can't read data
    major: Dataset
    minor: Read failed
  #005: /Users/ajelenak/Documents/h5py/hdf5/src/H5Dio.c line 401 in H5D__read(): can't read data
    major: Dataset
    minor: Read failed
  #006: /Users/ajelenak/Documents/h5py/hdf5/src/H5Dcontig.c line 842 in H5D__contig_read(): contiguous read failed
    major: Dataset
    minor: Read failed
  #007: /Users/ajelenak/Documents/h5py/hdf5/src/H5Dselect.c line 459 in H5D__select_read(): read error
    major: Dataspace
    minor: Read failed
  #008: /Users/ajelenak/Documents/h5py/hdf5/src/H5Dselect.c line 219 in H5D__select_io(): read error
    major: Dataspace
    minor: Read failed
  #009: /Users/ajelenak/Documents/h5py/hdf5/src/H5Defl.c line 453 in H5D__efl_readvv(): can't perform vectorized EFL read
    major: Dataset
    minor: Can't operate on object
  #010: /Users/ajelenak/Documents/h5py/hdf5/src/H5VM.c line 1263 in H5VM_opvv(): can't perform operation
    major: Internal error (too specific to document in detail)
    minor: Can't operate on object
  #011: /Users/ajelenak/Documents/h5py/hdf5/src/H5Defl.c line 403 in H5D__efl_readvv_cb(): EFL read failed
    major: Dataset
    minor: Read failed
  #012: /Users/ajelenak/Documents/h5py/hdf5/src/H5Defl.c line 276 in H5D__efl_read(): read error in external raw data file
    major: External file list
    minor: Read failed