Add WARPLossLayer and gradient check test cases

[kloudkl]

This pull request chose to implement the Weighted Approximate Rank Pairwise (WARP) loss layer in response to issue #88 "Implement ranking loss layer". WARP[1] was applied in a similar manner for image annotation[2] and is also useful in image retrieval using text queries.

The only open source implementation of WARP loss that I found is in the mrec recommender systems Python library from Mendeley. WARP was used there to optimize a matrix factorization model and a hybrid ranking model.

Obeying the contributing protocol designed by @shelhamer in #101, I opened this work in progress PR to welcome comments to correct defects as early as possible and to guide my further development.

References:
[1] Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up to large vocabulary image annotation. In IJCAI, 2011.
[2] Yunchao Gong, Yangqing Jia, Sergey Ioffe, Alexander Toshev, Thomas Leung. Deep Convolutional Ranking for Multilabel Image Annotation. arXiv:1312.4894 [cs.CV]

[kloudkl]

The base branch was not set to the newly created dev branch. The project maintainer can change it with the Pull Request API. Thanks!

curl --user ":owner" \
       --request POST \
       --data '{"issue": "126", "head": "kloudkl:warp_loss_layer", "base": "dev"}' \
       https://api.github.com/repos/BVLC/caffe/pulls/

[shelhamer]

As far as I know the API actually creates a new PR rather than updating the existing request. Since dev is a recent invention for Caffe this PR and other open PRs have amnesty, and we'll merge them to dev behind-the-scenes.

[shelhamer]

As to the actual content of this PR: ranking loss has been on my todo list as well, and WARP is a welcome choice of loss.

[sergeyk]

Thanks @kloudkl!
@shelhamer will review this after March 7.

[kloudkl]

I did not make progress since the opening of this PR because there is an assumption that was not satisfied, i.e. computing the loss relied on how multiple labels of a data point is store in the bottom label blob. Currently the DataLayer only supports single label input data. So the loss layer is blocked by the data layer. That's why I opened #144 asking for an external potential solution.

Now, the related works are concentrated in #149 thanks to @sergeyk. I will have discussions there to have the data layer implemented first and refine the WARP loss layer at the same time. Making changes to be coordinated with #149 is expected.

[sergeyk]

@kloudkl, if you need a multi-label input to make progress on this, the HDF5DataLayer provides both data and label as vectors. See test example.

[kloudkl]

@sergeyk, thanks for the HDF5DataLayer! I will give it a try. We also need tools to convert existing datasets into the HDF5 format.

[jay2002]

@kloudkl have you tried to convert your dataset into HDF5 format?
If my dataset have 4 (0, 1, 2, 3) labels and one sample has 3 labels (0, 1, 2)
should the label vetcor be [1 1 1 0]? or [0 1 2]?

[kloudkl]

@jay2002, I am too busy at some other related issues #244, #250 and #251 at the moment. So I haven't converted a dataset yet.

#220 used the C++ interface of HDF5 which must be manually compiled on Ubuntu 12.04. Have a look at it to see it helps. I will implement #213 using the C interface very soon.

@sergeyk, as the creator of HDF5DataLayer, you must have a lot of first hand experiences in doing so. How did you handle such situations?

[sergeyk]

ALthough HDF5DataLayer can load matrices for label data, I haven't ever
used that functionality. The answer to @jay2002's question depends entirely
on the loss layer. -- and I'm not up to date on our multi-label loss
options, sorry.

One thing I can say is that each row of the label matrix must be the same
size, so your option 2 doesn't make sense.

On Sun, Mar 23, 2014 at 12:35 AM, kloudkl notifications@github.com wrote:

@jay2002 https://github.com/jay2002, I am too busy at some other
related issues #244 #244, #250https://github.com/BVLC/caffe/pull/250and
#251 #251 at the moment. #213https://github.com/BVLC/caffe/issues/213will be the next target. So I havn't got the time to do the conversion.

@sergeyk https://github.com/sergeyk, as the creator of HDF5DataLayer,
you must have a lot of first hand experiences in doing so. How did you
handle such situations?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/126#issuecomment-38376170
.

[kloudkl]

HDF5OutputLayer has been implemented in #252. You can convert a dataset using a network with only a DataLayer and an HDF5OutputLayer.

[jay2002]

I got it. Thx @kloudkl @sergeyk

[kloudkl]

Continued in #257.