Feature request: native complex number datatype

[ZedThree]

Is your feature request related to a problem? Please describe.

Complex numbers are widely used in the physical sciences, and in some domains are very commonplace. C, C++, Fortran, Python, and Julia all have native complex numbers, all with the same representation in memory, that of an array of two numbers in the rectangular/Cartesian form. Currently, because there is no built-in datatype for HDF5 then library and application authors must create their own. A nice summary of the approaches different authors have taken is in this discussion on the CF-Conventions. Note that there are broadly speaking three approaches, but the one labelled 2a appears to be the most common.

Complex numbers have been discussed here previously, though still without real consensus:

• 2010 thread
• 2017 thread

Describe the solution you'd like

The minimal implementation that is most useful to the most number of people would be a set of complex types for float and double base types equivalent to the compound type:

DATATYPE "H5T_COMPLEX_<base float>" H5T_COMPOUND {
   H5T_<base float> "r";
   H5T_<base float> "i";
}

This would coincide with the choices made by h5py and HDF5.jl, for example.

The <base float> above would either be NATIVE_FLOAT/NATIVE_DOUBLE and/or the IEEE_F* types.

Describe alternatives you've considered

The existing solution is for each library and application to invent their own datatype. This has led to incompatible definitions for the type, reducing interoperability between tools and datasets. It also adds friction for developers and users in having to create the type or re-structure their data in comparison to other native types.

Additional context

Some commentators have called for this type to be flexible enough to work for higher order complex numbers such as quarternions. I'm not proposing that here as these types are generally not built-in types in the programming languages, at least not those used in the physical sciences. I believe that there may also be much more variety in the representations of quarternions so that it perhaps makes less sense to bless one particular form.

[Dave-Allured]

I fully support this proposal. Complex types are much overdue.

From numerous past discussions, I forsee a need for several predefined, "standard" types. It would be efficient to add all of these in a single push. I considered putting _IEEE_ within these suggested names, but left it out for compactness. The NATIVE types follow from these "standard" types, so I left those out, too.

H5T_COMPLEX_F16BE
H5T_COMPLEX_F16LE
H5T_COMPLEX_F32BE
H5T_COMPLEX_F32LE
H5T_COMPLEX_F64BE
H5T_COMPLEX_F64LE

H5T_POLAR_F16BE
H5T_POLAR_F16LE
H5T_POLAR_F32BE
H5T_POLAR_F32LE
H5T_POLAR_F64BE
H5T_POLAR_F64LE

COMPLEX = cartesian; real/imaginary, X, Y or what have you. POLAR = radial; R-theta, angular, or whatever.

[ZedThree]

Just to note that there aren't built-in types for the 16-bit floating types, so they'd probably be out of scope for complex numbers too.

I also wasn't proposing the polar versions because the language types (C, C++, Fortran, Python) are all stored in memory as the cartesian/rectangular form, and it's trivial to convert to/from polar. This conversion could be done as filter, for example. If the polar forms were added, then the conversion would have to be done to return the language native type.

[jwsblokland]

In the past, we also struggled to implement proper support of complex number until I realized the following: what if you extend the database by adding another dimension which represents the parts of a complex number. This additional dimension should be the fast varying one. In C ordering, for a three dimensional dataset of complex numbers of nx=7, ny=5 and nz = 3, it become like this:

DATASPACE  SIMPLE { ( 7, 5, 3, 2 ) / ( 7, 5, 3, 2 ) }

At least in C, C++ and Fortan an array of complex numbers are stored like: Re Im Re Im.... The main advantage of this approach, the HDF5 library does not have to be changed and you get all the dataset features, like type conversion, sub-selection, data transform functions, etc. for free. Writing and reading an array complex numbers is straightforward at least in C and C++. Another advantage this approach can be easily extended to store like quarternions or other higher order complex numbers.

[ZedThree]

Indeed, this is an alternative convention for storing complex numbers in HDF5 -- and actually this is one of the reasons for my (reintroduction) proposal. Because there is no native HDF5 complex datatype, there are multiple different conventions for storing complex numbers in the wild, which makes interoperability more difficult, and also makes it harder for developers.

What I think would be most useful is for there to be the built-in ability to use native language complex types with HDF5.

[ajelenak]

I received this morning a number of GitHub posts from the CF metadata conventions' repo about this issue. So it seems a critical mass has been reached in the CF-netCDF community that wants to see some implementation of this. Since this is a fairly large community, their decision how to store complex numbers in netCDF-4 files will have impact on whatever ends up the HDF5 choice.

Proposed are two approaches:

1. adding complex datatypes in libhdf5;
2. define a compound datatype and develop a convention how to describe them in HDF5 files.

The advantage of the second option is that it can immediately be implemented. There is also a precedent of using already available HDF5 features to support netCDF: HDF5 dimension scales were developed specifically to store netCDF dimensions in netCDF-4 files.

[Dave-Allured]

@ajelenak, I believe that built-in complex types are the ultimate solution and in the long run, will be well worth development effort. HDF5 built-ins will provide optimal solutions for standardization, language and platform compatibility, compactness, atomicity, parallel computing, user friendliness, and ending debate. Please notice that many scientific languages from the most classic (Fortran) to modern (Python), have adopted native complex types at some point in their history. This issue far transcends the CF and netCDF communities.

[jhendersonHDF]

Hi All,

we have an RFC now for adding support for complex number datatypes, as well as datatypes for 16-bit floating point values and some other useful datatypes. If you have the time, it would be appreciated if you could take a look at the RFC at https://forum.hdfgroup.org/t/hdf5-rfc-adding-support-for-16-bit-floating-point-and-complex-number-datatypes-to-hdf5/11975 and give any feedback you may have here or in that forum thread. Thanks!

[ajelenak]

The PR for complex number datatypes is out: #4630.