HDF5DataLayer: read matrix of features and labels from HDF5 file as input

[sergeyk]

This is useful such that we can use Caffe for classification purposes -- for this use case, we don't usually need the image-specific considerations that DataLayer is full of.

This adds a dependency on HDF5.

Future commits will add ability to read HDF5 files partially, and to iterate over multiple files in a directory.

[sergeyk]

Gonna merge this, @kloudkl's suggestion to also use this layer as the first example for #148 will come in a separate PR.

[sguada]

Need to update the Caffe Installation to mention which hdf5 package need to be installed like:
sudo apt-get install libhdf5-serial-dev

[sguada]

@sergeyk Actually I cannot compile this version, could you tell me which libraries I need to install?

[shelhamer]

He almost certainly linked against anaconda.

[sergeyk]

Nope, I'm running all homebrew python. It's simply brew install hdf5 on
Mac. Presumably apt-get install libhdf5-dev.

On Tuesday, February 25, 2014, Evan Shelhamer notifications@github.com
wrote:

He almost certainly linked against anaconda.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/147#issuecomment-36083444
.

[sguada]

I tried apt-get install libhdf5-serial-dev and still doesn't work, at least in ubuntu 12.04

/usr/bin/g++ build/examples/demo_compute_image_mean.o libcaffe.a -o build/examples/demo_compute_image_mean.bin -pthread -fPIC -DNDEBUG -O2 -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I./src -I./include -I/usr/local/cuda/include -I/opt/intel/mkl/include -L/usr/local/lib -L/usr/lib -L/usr/local/lib -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib -L/opt/intel/mkl/lib -L/opt/intel/mkl/lib/intel64 -lcudart -lcublas -lcurand -lmkl_rt -lpthread -lglog -lprotobuf -lleveldb -lsnappy -lboost_system -lhdf5 -lhdf5_hl -lopencv_core -lopencv_highgui -lopencv_imgproc -Wall
/usr/lib/libhdf5_hl.so: undefined reference to `H5Pcopy'
/usr/lib/libhdf5_hl.so: undefined reference to `H5T_STD_U16LE_g'
/usr/lib/libhdf5_hl.so: undefined reference to `H5Oget_info'
/usr/lib/libhdf5_hl.so: undefined reference to `H5Pset_preserve'
/usr/lib/libhdf5_hl.so: undefined reference to `H5Dset_extent'
/usr/lib/libhdf5_hl.so: undefined reference to `H5Pcreate'

[sergeyk]

Not sure what to tell you.
I'm running Ubuntu 12.04 as well, and it works fine for me, here's my log:

caffe [dev] % make
protoc --proto_path=src --cpp_out=src src/caffe/proto/caffe.proto
mkdir -p include/caffe/proto
cp src/caffe/proto/caffe.pb.h include/caffe/proto/

/usr/bin/g++ src/caffe/proto/caffe.pb.cc -pthread -fPIC -DNDEBUG -O2 -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I./src -I./include -I/usr/local/cuda/include -I/opt/intel/mkl/include -c -o build/src/caffe/proto/caffe.pb.o

/usr/bin/g++ src/caffe/util/insert_splits.cpp -pthread -fPIC -DNDEBUG -O2 -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I./src -I./include -I/usr/local/cuda/include -I/opt/intel/mkl/include -c -o build/src/caffe/util/insert_splits.o

/usr/bin/g++ src/caffe/util/math_functions.cpp -pthread -fPIC -DNDEBUG -O2 -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I./src -I./include -I/usr/local/cuda/include -I/opt/intel/mkl/include -c -o build/src/caffe/util/math_functions.o

/usr/bin/g++ src/caffe/util/io.cpp -pthread -fPIC -DNDEBUG -O2 -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I./src -I./include -I/usr/local/cuda/include -I/opt/intel/mkl/include -c -o build/src/caffe/util/io.o
In file included from src/caffe/util/io.cpp:19:0:
./include/caffe/util/io.hpp:9:18: fatal error: hdf5.h: No such file or directory
compilation terminated.
make: *** [build/src/caffe/util/io.o] Error 1
caffe [dev] % sudo apt-get install libhdf5-dev
[sudo] password for sergeyk:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package libhdf5-dev is a virtual package provided by:
  libhdf5-serial-dev 1.8.4-patch1-3ubuntu2
  libhdf5-openmpi-dev 1.8.4-patch1-3ubuntu2
  libhdf5-mpich-dev 1.8.4-patch1-3ubuntu2
  libhdf5-lam-dev 1.8.4-patch1-3ubuntu2
You should explicitly select one to install.

E: Package 'libhdf5-dev' has no installation candidate
caffe [dev] % sudo apt-get install libhdf5-serial-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  gir1.2-ubuntuoneui-3.0 libubuntuoneui-3.0-1
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libgfortran3 libhdf5-serial-1.8.4
Suggested packages:
  libhdf5-doc
The following NEW packages will be installed:
  libgfortran3 libhdf5-serial-1.8.4 libhdf5-serial-dev
0 upgraded, 3 newly installed, 0 to remove and 15 not upgraded.
Need to get 4,043 kB of archives.
After this operation, 22.7 MB of additional disk space will be used.
Do you want to continue [Y/n]?
Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main libgfortran3 amd64 4.6.3-1ubuntu5 [357 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/universe libhdf5-serial-1.8.4 amd64 1.8.4-patch1-3ubuntu2 [1,408 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/universe libhdf5-serial-dev amd64 1.8.4-patch1-3ubuntu2 [2,278 kB]
Fetched 4,043 kB in 10s (373 kB/s)
Selecting previously unselected package libgfortran3.
(Reading database ... 201375 files and directories currently installed.)
Unpacking libgfortran3 (from .../libgfortran3_4.6.3-1ubuntu5_amd64.deb) ...
Selecting previously unselected package libhdf5-serial-1.8.4.
Unpacking libhdf5-serial-1.8.4 (from .../libhdf5-serial-1.8.4_1.8.4-patch1-3ubuntu2_amd64.deb) ...
Selecting previously unselected package libhdf5-serial-dev.
Unpacking libhdf5-serial-dev (from .../libhdf5-serial-dev_1.8.4-patch1-3ubuntu2_amd64.deb) ...
Setting up libgfortran3 (4.6.3-1ubuntu5) ...
Setting up libhdf5-serial-1.8.4 (1.8.4-patch1-3ubuntu2) ...
Setting up libhdf5-serial-dev (1.8.4-patch1-3ubuntu2) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
caffe [dev] % make
/usr/bin/g++ src/caffe/util/io.cpp -pthread -fPIC -DNDEBUG -O2 -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I./src -I./include -I/usr/local/cuda/include -I/opt/intel/mkl/include -c -o build/src/caffe/util/io.o

/usr/bin/g++ src/caffe/util/im2col.cpp -pthread -fPIC -DNDEBUG -O2 -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I./src -I./include -I/usr/local/cuda/include -I/opt/intel/mkl/include -c -o build/src/caffe/util/im2col.o

and so on

[sergeyk]

Actually, I gey these linker errors too, wonder what's up

/usr/lib/libhdf5_hl.so: undefined reference to `H5Pcopy'

[sguada]

That's the error I get

Sergio

2014-02-25 19:25 GMT-08:00 Sergey Karayev notifications@github.com:

Actually, I gey these linker errors too, wonder what's up

/usr/lib/libhdf5_hl.so: undefined reference to `H5Pcopy'

Reply to this email directly or view it on GitHubhttps://github.com//pull/147#issuecomment-36087192
.

[shelhamer]

Check the order of the includes and library paths in the makefile. You could be building against the anaconda version but linking against the /usr version or vice-versa.

I ran into something like this before.

[kloudkl]

The problem is solved in the commit 91eb77e "Reverse the order of hdf5_hl hdf5 as LIBRARIES in Makefile" in PR #159.

The library that is depended on should be placed after the depending ones.

[sergeyk]

Let's merge in your PR quickly then.

[sguada]

@kloudkl good catch. Now it compiles in Ubuntu too @sergeyk @shelhamer

Cheers!!