Multi-class object detection

[gheinrich]

• Update Python clustering layers to output one top per class
• Add example model for 2-class object detection

[lukeyeager]

This looks good and doesn't appear to break existing networks / workflows.

[lukeyeager]

Together with NVIDIA/DIGITS#869, I can train this network and verify bounding boxes for multiple classes.

[szm-R]

Could it be used for detecting more than two classes? and what about the object sizes? I mean according to Deep Learning for Object Detection with DIGITS "the object size should be between 50×50 and 400×400 px in the input images" so trying to detect multiple objects with different sizes could lead to problems? also related to the questions I have asked in BVLC#1011

[szm-R]

Hi, in detectnet_network-2classes.prototxt what does src and dst mean in these lines (82 and 83)?
object_class: { src: 1 dst: 0} # cars -> 0
object_class: { src: 8 dst: 1} # pedestrians -> 1

[gheinrich]

src is the class ID (see doc - by default 1 is for car and 8 is for pedestrian). dst is where the corresponding labels are mapped in the output of the data transformation layer. It might be easier to understand if you try and compare the net descriptions for one class / two classes.

[szm-R]

Thank you, I tried to duplicate the procedure with my data setting the classes to:
object_class: { src: 4 dst: 0}
object_class: { src: 3 dst: 1}
Everything else is the same except image sizes
The data actually contains three labels but here I only want to use two of them, however when I run the model I get this error (from caffe output):

I0830 19:00:35.659642 21846 net.cpp:94] Creating Layer score-class1
I0830 19:00:35.659647 21846 net.cpp:435] score-class1 <- bbox-list-label-class1
I0830 19:00:35.659651 21846 net.cpp:435] score-class1 <- bbox-list-class1
I0830 19:00:35.659657 21846 net.cpp:409] score-class1 -> bbox-list-scored-class1
F0830 19:00:35.659672 21846 blob.cpp:113] Check failed: data_

I checked the log and found out that nothing is created for the second class in previous layers:
I0830 19:00:35.638110 21846 net.cpp:94] Creating Layer cluster
I0830 19:00:35.638131 21846 net.cpp:435] cluster <- coverage_coverage/sig_0_split_1
I0830 19:00:35.638139 21846 net.cpp:435] cluster <- bboxes_bbox/regressor_0_split_1
I0830 19:00:35.638146 21846 net.cpp:409] cluster -> bbox-list-class0
I0830 19:00:35.638155 21846 net.cpp:409] cluster -> bbox-list-class1
I0830 19:00:35.638284 21846 net.cpp:144] Setting up cluster
I0830 19:00:35.638293 21846 net.cpp:151] Top shape: 5 50 5 (1250)
I0830 19:00:35.638298 21846 net.cpp:151] Top shape: (0)
I0830 19:00:35.638300 21846 net.cpp:159] Memory required for data: 3235175888
I0830 19:00:35.638304 21846 layer_factory.hpp:77] Creating layer cluster_gt
I0830 19:00:35.638329 21846 net.cpp:94] Creating Layer cluster_gt
I0830 19:00:35.638334 21846 net.cpp:435] cluster_gt <- coverage-label_slice-label_4_split_1
I0830 19:00:35.638339 21846 net.cpp:435] cluster_gt <- bbox-label_slice-label_1_split_1
I0830 19:00:35.638345 21846 net.cpp:409] cluster_gt -> bbox-list-label-class0
I0830 19:00:35.638352 21846 net.cpp:409] cluster_gt -> bbox-list-label-class1
I0830 19:00:35.638432 21846 net.cpp:144] Setting up cluster_gt
I0830 19:00:35.638438 21846 net.cpp:151] Top shape: 5 50 5 (1250)
I0830 19:00:35.638442 21846 net.cpp:151] Top shape: (0)

What am I missing?! I created the db leaving the custom classes empty (to use the default mapping) and here is the output:
Entry Count
998
Feature shape
(3, 608, 800)
Label shape
(1, 11, 16)

I also don't understand what are the numbers in Label shape. could it be possible that the db process has not included the other labels?! my label files look sth like this:
Bus 0.0 0 0.0 0.00 68.00 78.00 162.00 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Truck 0.0 0 0.0 60.00 91.00 150.00 190.00 0.0 0.0 0.0 0.0 0.0 0.0 0.0

(different combinations of the three labels I have can be found in various images)

[gheinrich]

Can you check that you're using a version of Caffe that includes this PR?

[szm-R]

Sorry how can I check that? and also I don't understand what you mean by "this PR"

[szm-R]

Ok, in info (in the GUI) the caffe version is: 0.15.9

[gheinrich]

Sorry I hate it when people use acronyms and I'm doing it :-( . On Github "PR" stands for "Pull Request". What I meant to ask was whether your code had the changes from this patch. This patch was merged just after 0.15.9. To get those changes you will need to build Caffe from source. You may follow this guide.

[szm-R]

OK, I built caffe from source using the guide you said and it finished without any errors. now should I do anything for DIGITS to see this caffe? (I set the CAFFE_HOME environment variable to my caffe home as the guide suggested) The thing is I'm still getting the same error for two class detection.

[gheinrich]

You will need to edit the DIGITS config to point to your own Caffe binary. Browse the doc to find out how to do this.

[szm-R]

Thank you I did it and the problem resolved after a reboot, I have now successfully launched it for a 3 class detection, thank you again for your patience and your quick answers.

[szm-R]

Hi, in 3 class training, the network is showing around 70 percent mAP for one class and 0 for the other two after epoch 46, Is there sth wrong what I've done? or simply I should wait for it to go further?

[gheinrich]

Hello, multi-class object detection with DetectNet is provided for adventurers who would like to experiment. We don't have a definite guide to training these models as the choice of parameters is very much dataset specific. It should help if your dataset is well balanced. You may also try to incrementally increase the number of classes (e.g. fine-tune a single-class model to do dual-class detection).

[szm-R]

Thank you

On Fri, Sep 2, 2016 at 5:10 PM, Greg Heinrich notifications@github.com
wrote:

Hello, multi-class object detection with DetectNet is provided for
adventurers who would like to experiment. We don't have a definite guide to
training these models as the choice of parameters is very much dataset
specific. It should help if your dataset is well balanced. You may also try
to incrementally increase the number of classes (e.g. fine-tune a
single-class model to do dual-class detection).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#157 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/APaJX6q-xgrzq2KNq_Z0z3mWv6Yc2tO2ks5qmBk0gaJpZM4IssbU
.

[xhuvom]

Hi, Im using DIGITS (5.1-dev) Detectnet for multiclass classification. My caffe version is 0.15.14. Single class detection works fine. I have trained Detectnet for multiclass classification (cars & pedestrian) with the prototext pulled in this thread. The rest of the settings remained unchanged as the walkthrough . After 30 epoch, the training mAP values for both classes remains zero. It can't inference anything at all. Should I proceed with further epochs for a reasonable inferencing result?

[xhuvom]

@gheinrich @lukeyeager
After about 100 epochs, still the mAP value for second class remains zero. Although several restarts from previous models occurred, hope it doesn't affect overall training performance. I wonder what is going wrong with detecting second class? Also the mAP value for 1st class remains low compared to single object detection model. The current training state (after about 100 epochs) looks like this-

[ghost]

@gheinrich @lukeyeager @szm2015 @xhuvom
I am trying to train a model for car & pedestrian using detectnet _network-2classes.prototxt. for kiiti data. the model is trained for 250 Epochs with basic learning rate of 0.0001 & layer learning rate 0.1 . but still mAP for both the classes are still 0. can you guide be where I might have gone wrong, or what changes Need to be done ? the configurations are as :
UBUNTU 14.04
Caffe > 0.15.9
GPU:- TX980 Ti

[xhuvom]

I also had the same problem for multiclass detection. Anyways, I found YOLO Darknet useful for detection of many class at high fps. U might check it out at http://pjreddie.com/darknet/yolo/

[ShashankVBhat]

@gheinrich @lukeyeager @szm2015 @xhuvom
When I trained on KittiDataset for car using DetectNet, I got mAP after 1 epoch & got good results.
But now I am trying to train for human dataset (own dataset) using DetectNet it is not converging (mAP=0) even after 60 epochs details of the training are as follow:'
Training Images=2656, Validation Images=952
Resize image = 1248 * 384 (Since Datasets were of different size)
Solver = SGD
Basic Learning Rate = 0.001 (Fixed)
Batch Size = 4
Batch Accumulation =6
Weights= GoogleNet pre-trained weights
Layer learning rate of weights = 0.001
GPU GTX980 Ti

I just wanted to know whether resize of images will affect the bounding box coordinates in the label file or DIGITS internally modifies it ????
Is it not converging because i have reduced the learning rate??
Should I train it for some more epochs???

Can you guide me to get good results.

[ShervinAr]

@gheinrich @lukeyeager
Hi there. Is there any chance you could provide the caffe detectnet model of DIGITS trained for multi-class detection of car&pedestrians?
I would like to initialize weights of the network with something meaningful for the detection of both classes and then proceed with fine-tuning the weights ...
many thanks