Refactor data_transform to allow datum, cv:Mat and Blob transformation #1070

sguada · 2014-09-11T21:57:06Z

Move data_mean into the data_transform class to facilitate data transformation.
This way Data layers don't need to hold it and pass it all the time.

sguada · 2014-09-12T01:12:07Z

To prepare for transformation layers #569, modify #954 to manage the data_mean by the transformation.
Could you take a look @shelhamer @jeffdonahue?

kloudkl · 2014-09-12T04:49:01Z

src/caffe/layers/window_data_layer.cpp

@@ -161,6 +161,18 @@ void WindowDataLayer<Dtype>::DataLayerSetUp(const vector<Blob<Dtype>*>& bottom,
  // label
  (*top)[1]->Reshape(batch_size, 1, 1, 1);
  this->prefetch_label_.Reshape(batch_size, 1, 1, 1);
+
+  // data mean
+  if (this->layer_param_.window_data_param().has_mean_file()) {


Why should the WindowDataLayer be handled specially?

Because the transformation done by WindowDataLayer is more complicated and specific to that layer.
But it could be abstracted later on.

The WindowDataLayer is quite specific and can be left out of standardization for now in how it's coded. It was annoying enough to make me defer data layer factoring before. However, it should have the same transform_param interface this->layer_param_.transform_param().has_mean_file() since there's no reason for the user to know the difference when defining their model.

jeffdonahue · 2014-09-16T17:47:24Z

src/caffe/data_transformer.cpp

+  const int size = datum.channels() * datum.height() * datum.width();
+  if (data_mean_.count() < size) {
+    data_mean_.Reshape(1, datum.channels(), datum.height(), datum.width());
+    LOG(INFO) << "Transform without mean";


I don't think I understand what's going on here -- why reshape the mean inside Transform?

Just to cover the case where there was no mean_file in the parameters, and
therefore was not known the size a priori.

Sergio

2014-09-16 10:47 GMT-07:00 Jeff Donahue notifications@github.com:

In src/caffe/data_transformer.cpp:

ReadProtoFromBinaryFileOrDie(mean_file.c_str(), &blob_proto);

data_mean_.FromProto(blob_proto);

}
+}

+template
+void DataTransformer::Transform(const int batch_item_id,

const Datum& datum,

Dtype\* transformed_data) {

CHECK_GT(datum.channels(), 0);

CHECK_GE(datum.height(), param_.crop_size());

CHECK_GE(datum.height(), param_.crop_size());

const int size = datum.channels() * datum.height() * datum.width();

if (data_mean_.count() < size) {

data_mean_.Reshape(1, datum.channels(), datum.height(), datum.width());

LOG(INFO) << "Transform without mean";

I don't think I understand what's going on here -- why reshape the mean
inside Transform?

—
Reply to this email directly or view it on GitHub
https://github.com/BVLC/caffe/pull/1070/files#r17617821.

But Reshape doesn't actually imresize the mean, it will just zero it out, no? When might you want this behavior?

sguada · 2014-09-20T00:34:06Z

By the during my test I verified that LMDB is 10-15% faster than LEVELDB, however in most situations don't matter since the computation takes longer than prefetching.

jeffdonahue · 2014-09-20T01:12:36Z

@sguada let me know when you're done rebasing and I'll review it.

sguada · 2014-09-20T01:13:58Z

I think it's done and once Travis pass you can review it. Thanks

On Friday, September 19, 2014, Jeff Donahue notifications@github.com
wrote:

@sguada https://github.com/sguada let me know when you're done rebasing
and I'll review it.

—
Reply to this email directly or view it on GitHub
#1070 (comment).

Sergio

jeffdonahue · 2014-09-20T01:21:18Z

Er, I actually meant to post that on the matcaffe PR, but I'll take a look at this one as well.

bhack · 2014-09-20T07:12:07Z

@sguada Do you plan to remove cropping and mirroring from this code in #569? If not this code slowdown Mat copying especially for 1 channel continuous Mat. Using "at" addressing is the slowest method. @shelhamer I think this suffer of the OSX opencv cuda clang issue.

sguada · 2014-09-20T11:32:59Z

@bhack I think your optimization for continuous cv::Mat #1068 could be incorporated here while still allowing to do cropping and mirroring. Or it could be done in two steps, first transform cv::Mat into Blob and then use the Transform Blob to Blob to do that (although it would be a bit more inefficient maybe it would be cleaner).

sguada · 2014-09-20T11:44:46Z

src/caffe/util/benchmark.cpp

+  }
+  if (Caffe::mode() == Caffe::GPU) {
+#ifndef CPU_ONLY
+    CUDA_CHECK(cudaEventElapsedTime(&elapsed_microseconds_, start_gpu_,


cudaEventElapsedTime only measure milliseconds. Should divide by 1000 to have micro seconds

Is this note resolved elsewhere?

bhack · 2014-09-20T18:43:14Z

@sguada this template seems interesting also if this is not exactly the blob case but could be adapted.

BlGene · 2014-09-21T15:42:43Z

Hi Guys,

I added my transformation as opencv affine transforms. See here . This has the benefit of flexibly handling things like rotations that are hard to do other wise. If something like this is merge able in this context I would be happy to clean it up a bit.

BR, max

bhack · 2014-09-21T15:57:35Z

Could be interesting but we need to think how to organize and add multiple transformations and how to expose different transformation parameters in proto

BlGene · 2014-09-21T17:17:47Z

I would very much like prefetch_data_ to be promoted to a vector (see here). Maybe something that could be added here. In my cases it's so that I can output different scale images. Could that go here or should I make it a separate pull request.

bhack · 2014-09-21T17:56:37Z

We need to handle all the opencv transformations pipeline (when user specify a pipeline of serial transformation) before the Mat is copied inside the blob

sguada · 2014-09-21T18:09:22Z

@BlGene I think many transformation are easier done with cv::Mat, and could come later in another PR.
I think we should settle on the basic and then add more transformations later.

Here are some thoughts about it:

For current transformations, scale, mean subtraction, cropping and mirroring we can use the current code for datum.
For new transformation, rotation, scale, ... we could transform datum into cv::Mat and then apply those transformation, and then transform cv::Mat into Blob
To have multiple scale images, I would create another datalayer that have multiple tops and could have a vector of prefectch_data. But mixing this with current datalayers would be a bit difficult.

@bhack do you want to commit an optimized function to copy cv::Mat into Blob<Dtype? Or do you want me to extract that from your code? This could be a protected member of transform_data

shelhamer · 2014-09-21T18:13:55Z

@Bigene for multi-scale output / an image pyramid I took the approach of
keeping prefetch_data_ the same and then assigning each scale in the
pyramid its own top blob. That is, you can load the original image once and
then transform it as many times as you like and feed it into multiple tops.

On Sun, Sep 21, 2014 at 11:09 AM, Sergio Guadarrama <
notifications@github.com> wrote:

@BlGene https://github.com/BlGene I think many transformation are
easier done with cv::Mat, and could come later in another PR.
I think we should settle on the basic and then add more transformations
later.

Here are some thoughts about it:

For current transformations, scale, mean subtraction, cropping and
mirroring we can use the current code for datum.

For new transformation, rotation, scale, ... we could transform
datum into cv::Mat and then apply those transformation, and then transform
cv::Mat into Blob

To have multiple scale images, I would create another datalayer that
have multiple tops and could have a vector of prefectch_data. But mixing
this with current datalayers would be a bit difficult.

@bhack https://github.com/bhack do you want to commit an optimized
function to copy cv::Mat into Blob<Dtype? Or do you want me to extract that
from your code? This could be a protected member of transform_data

—
Reply to this email directly or view it on GitHub
#1070 (comment).

Make lint happy Conflicts: src/caffe/data_transformer.cpp

Conflicts: src/caffe/util/io.cpp

Added example of use to models/bvlc_reference_caffenet/train_val_mean_value.prototxt

Remove benchmarking code, fixed data_transformer doxygen docs

Make lint happy Fix Name conventions

sguada · 2014-10-03T23:43:44Z

@shelhamer ready for merge

shelhamer · 2014-10-04T01:10:59Z

Awesome, thanks Sergio! I made two tiny changes

Include the mean value example as comments in the standard CaffeNet prototxt. As a follow-up we could include it as a stage sometime (but I think the solver proto needs a train_stage first).
fix unit8 -> uint8 spelling

Refactor data_transform to allow datum, cv:Mat and Blob transformation

sguada · 2014-10-04T02:18:50Z

@shelhamer thanks for the review and the changes.

We probably want to move to mean_value models and forget about the mean_file.

shelhamer · 2014-10-04T03:15:21Z

@sguada agreed about the migration to mean values.

Refactor data_transform to allow datum, cv:Mat and Blob transformation

Mingcong · 2014-11-06T06:21:18Z

@sguada Could you give example about feed cvMat into Caffe？I plan to use Caffe to processing continuous streaming data, such as video.

sguada force-pushed the move_data_mean branch from d17d310 to 0c46c46 Compare September 11, 2014 23:02

kloudkl reviewed Sep 12, 2014
View reviewed changes

sguada mentioned this pull request Sep 16, 2014

set up datum size for WindowDataLayer #1091

Merged

jeffdonahue reviewed Sep 16, 2014
View reviewed changes

shelhamer force-pushed the dev branch from 64258b6 to 403b56b Compare September 19, 2014 04:38

sguada mentioned this pull request Sep 19, 2014

Re-add support to in memory Opencv io #1068

Closed

sguada force-pushed the move_data_mean branch from ac5e4d4 to 05e39e4 Compare September 19, 2014 21:57

sguada changed the title ~~Move data_mean into data_transformer~~ Refactor data_transform to allow datum, cv:Mat and Blob transformation Sep 19, 2014

sguada mentioned this pull request Sep 20, 2014

Fix types of SetUp, Forward, Backward, and gradient checker calls #945

Merged

sguada force-pushed the move_data_mean branch from fb85029 to 528c0e8 Compare September 20, 2014 00:41

sguada mentioned this pull request Sep 20, 2014

Transform layers [DON'T MERGE] #569

Closed

sguada reviewed Sep 20, 2014
View reviewed changes

sguada mentioned this pull request Sep 21, 2014

Conflicting Parameters in WindowDataLayer #1125

Closed

sguada added 5 commits October 3, 2014 11:45

Update description data_transformer.hpp

28ee58e

Refactor common code

d1ccbe3

Make lint happy Conflicts: src/caffe/data_transformer.cpp

Add flag check_size=false to convert_imageset

497eb60

Added test_io and faster cv::Mat processing

5527e89

Conflicts: src/caffe/util/io.cpp

Add CVMatToDatum

242546f

sguada force-pushed the move_data_mean branch from 24cea6b to 383fd35 Compare October 3, 2014 19:17

sguada added 6 commits October 3, 2014 16:42

Add ReadImageToDatumReference to test_io

408c258

Added more tests to test_io for CVMatToDatum

0a96030

Added mean_value to specify mean channel substraction

a9572b1

Added example of use to models/bvlc_reference_caffenet/train_val_mean_value.prototxt

Fix OSX compilation for nvcc with opencv

8cd863e

Fix calls to Rand() and test_data_layer error

8a6d348

Remove benchmarking code, fixed data_transformer doxygen docs

Fixed crop error and add test_data_transformer

b07a54f

Make lint happy Fix Name conventions

sguada force-pushed the move_data_mean branch from 383fd35 to b07a54f Compare October 3, 2014 23:43

shelhamer added 2 commits October 3, 2014 18:02

bundle pixel mean into CaffeNet as comments

3115b20

uin8 spell check

e8b48b4

sguada added a commit that referenced this pull request Oct 4, 2014

Merge pull request #1070 from sguada/move_data_mean

0ba046b

Refactor data_transform to allow datum, cv:Mat and Blob transformation

sguada merged commit 0ba046b into BVLC:dev Oct 4, 2014

sguada mentioned this pull request Oct 6, 2014

Aspect keeping resize in convert_imageset #1178

Closed

mlapin mentioned this pull request Oct 7, 2014

Legacy nvcc support #1236

Merged

shelhamer mentioned this pull request Oct 10, 2014

Opencv io support #1061

Closed

mitmul pushed a commit to mitmul/caffe that referenced this pull request Oct 11, 2014

Merge pull request BVLC#1070 from sguada/move_data_mean

31fa1eb

Refactor data_transform to allow datum, cv:Mat and Blob transformation

sguada mentioned this pull request Oct 13, 2014

Segmentation Fault with hybrid Overfeat-Alexnet architecture for large input images #1260

Closed

bhack mentioned this pull request Nov 1, 2014

Data augmentation via random rotations #1386

Closed

RazvanRanca pushed a commit to RazvanRanca/caffe that referenced this pull request Nov 4, 2014

Merge pull request BVLC#1070 from sguada/move_data_mean

157aee9

Refactor data_transform to allow datum, cv:Mat and Blob transformation

sguada deleted the move_data_mean branch November 28, 2014 11:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor data_transform to allow datum, cv:Mat and Blob transformation #1070

Refactor data_transform to allow datum, cv:Mat and Blob transformation #1070

sguada commented Sep 11, 2014

sguada commented Sep 12, 2014

kloudkl Sep 12, 2014

sguada Sep 12, 2014

shelhamer Sep 14, 2014

jeffdonahue Sep 16, 2014

sguada Sep 16, 2014

jeffdonahue Sep 16, 2014

sguada commented Sep 20, 2014

jeffdonahue commented Sep 20, 2014

sguada commented Sep 20, 2014

jeffdonahue commented Sep 20, 2014

bhack commented Sep 20, 2014

sguada commented Sep 20, 2014

sguada Sep 20, 2014

shelhamer Oct 2, 2014

bhack commented Sep 20, 2014

BlGene commented Sep 21, 2014

bhack commented Sep 21, 2014

BlGene commented Sep 21, 2014

bhack commented Sep 21, 2014

sguada commented Sep 21, 2014

shelhamer commented Sep 21, 2014

sguada commented Oct 3, 2014

shelhamer commented Oct 4, 2014

sguada commented Oct 4, 2014

shelhamer commented Oct 4, 2014

Mingcong commented Nov 6, 2014

Refactor data_transform to allow datum, cv:Mat and Blob transformation #1070

Refactor data_transform to allow datum, cv:Mat and Blob transformation #1070

Conversation

sguada commented Sep 11, 2014

sguada commented Sep 12, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sguada commented Sep 20, 2014

jeffdonahue commented Sep 20, 2014

sguada commented Sep 20, 2014

jeffdonahue commented Sep 20, 2014

bhack commented Sep 20, 2014

sguada commented Sep 20, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bhack commented Sep 20, 2014

BlGene commented Sep 21, 2014

bhack commented Sep 21, 2014

BlGene commented Sep 21, 2014

bhack commented Sep 21, 2014

sguada commented Sep 21, 2014

shelhamer commented Sep 21, 2014

sguada commented Oct 3, 2014

shelhamer commented Oct 4, 2014

sguada commented Oct 4, 2014

shelhamer commented Oct 4, 2014

Mingcong commented Nov 6, 2014