HDF

Description

• Current version: 5
• Binary file format
• Usual extensions:
  • .h5
  • .hdf5
  • .hdf
• Groups:
  • Specify location of data
  • Can be compared to directory/dictionary keys
• Data Sets:
  • Store data
  • Can be compared to multidimensional arrays/tables

Information

• Learning HDF5

Libraries

• h5py

  • Mapping from HDF5 library to Python and numpy
  • Seems to be a little slower than PyTables
  • Size of data is fixed and can be resized later up to a fixed chosen maximum
    • This should not be a big problem since storing a large amount of data with value 0 still results in a 1 KB file

• PyTables

  • Database like interface to HDF5
  • Installation requires HDF5 C library or binary wheel file (Windows)

Tools

• h5dump
  • Prints data contained in an HDF5 file
  • Contained in HDF5 software distribution (macOS)
• ViTables
  • Graphical viewer and editor for HDF5 files
• HDF View
  • Official graphical viewer and editor for HDF files

Comparison

Data sizes for 60 seconds of measurement data containing:

• message counter
• time stamp
• acceleration value

Format	Size (MB)	Works in HDFView
Log Data	41.929378
CSV	13.388231
PyTables No Compression	6.299136	✅
h5py Zfp	6.262768	✅
h5py FciDecomp	6.262760	✅
h5py No Compression	6.250103	✅
h5py LZF	3.360693	❌
h5py LZ4	2.998656	❌
h5py Blosc	1.913076	❌
h5py GZip	1.731552	✅
h5py Bitshuffle	1.709330	❌
PyTables lzo	1.630033	❌
PyTables blosc	1.540903	❌
h5py Zstd	1.388591	❌
PyTables zlib	1.137004	✅

Tool

For the data above we used the Python script convert.py:

python Source/convert.py

The script will store the output files in the directory Data. Afterward we used the Unix command:

ls -lS Data/*

to print the size of the converted files.

Requirements

To use the script you need Python 3.9 or later and the

• h5py,
• tables, and
• hdf5plugin

packages:

pip install h5py hdf5plugin tables