Util functions converting formats between HDF5 and Blob #220
Conversation
|
The direct purpose of this PR is to convert the features extracted by #161 into HDF5 for loading back conveniently later. The dims are assumed to be four but can be relaxed easily. @sergeyk, your #203 combined with @tdomhan's #217 provided similar IO utilties and a customized data layer. I should have a look at whether there have been similar efforts before coding. It is the first time here that multiple people worked on so closely related issues almost at the same time. Please continue to refine your PR. |
|
If #203 does not intend to save Blobs into the HDF5 format files, then this PR still has something useful. |
|
@kloudkl I will finish up my PR and @sguada will merge it into dev. This PR is still valuable, because writing out to HDF5 is on our todo list: #213. My main concern is adding the dependency on the HDF5 C++ interface. I see that it facilitates copying data. On the other hand, I think the solution in #203 does a fine job with only the C interface. Pinging @sguada on this one. I think that the C++ interface will eventually be useful to us, for example for reading chunks at a time. For now, my instinct is to leave this PR dormant, and ask @kloudkl to submit a PR for #213, and to only use the C interface if possible. if @kloudkl finds that #213 is made significantly easier/cleaner by the C++ interface, then I would be for including it as part of the Caffe source code to make compilation easier. |
|
@kloudkl shall we close this PR? |
Fix for broken DataLayerTest/2.TestReshapeLevelDB test
To use the HDF5DataLayer to provide advanced data set and to enable input from various data sources, there should be functions converting Blob from and to other formats.
HDF5 is a very popular data set format in the scientific computing community. Building the bridge between it and Blob will open a lot of new opportunities to solve #126 and many complex practical problems.