how to Parallel /cocurrent read a same file?

[algoqt]

I have a H5 file , when I start some diffent thread to read different dataset in it , it still queueing read, a thread read done then another thread , this cause a lot of time of waiting , how to turn on Parallel /cocurrent read .

HighFive::File pfile(h5file, HighFive::File::ReadOnly);

[algoqt]

I google quit a time but seems it is not esay to sovle this problem?

[1uc]

You need to distinguish between two types of parallel: MPI-parallel, i.e. distinct processes, and multi-threaded parallelism.

HDF5 is very good at the former, i.e. MPI-based parallelism has been supported for a very long time and is tuned towards the common parallel filesystems found in supercomputers.

The later, i.e. multi-threaded parallelism, is not officially supported. For a generic file, the only thing I know that can (if lucky) be done with reasonable effort is making it thread-safe, by using a lock to prevent all concurrency of calls to HDF5 (or using the thread-safe version of the library). For simple files and if you pay close attention to all the warnings, you can use H5Dread_chunk to read a dataset as a sequence of unprocessed bytes into RAM:
https://docs.hdfgroup.org/hdf5/v1_12/group___h5_d.html#gac1092a63b718ec949d6539590a914b60
The interesting feature of H5Dread_chunk is that it's concurrent and thread-safe. Note, that none of the other HDF5 functions you'll need to call before using H5Dread_chunk are, so those need to be protected against concurrent access, e.g. with a mutex.

Remember that often disk access is, by orders of magnitude, slower than RAM. Unless you have specialized hardware, the other threads would always end up waiting anyway, because the disk is already busy serving the first thread.

Since it might apply, HDF5 is by default rather slow when using many small datasets. Here small means datasets with maybe less than 1e6 elements; and many however many you need to make I/O an bottleneck in your application. Techniques for fusing datasets might be a more sustainable choice, if you're able to change the file layout of the files you need to read.