Add RetinaNet Implementation

[chengyangfu]

Hi,
This PR contains the RetinaNet implementation. The following table contains the models which use ResNet50 and ResNet101 as the backbones.

GPU : Pascal Titan X
PyTorch : v1.0.0
Inference time is measured when setting batch size as 1.

Model	Detectron Accuracy	Current Accuracy	inference time(s/im)	download
RetinaNet_R-50-FPN_1x	35.7	36.3	0.102	model
RetinaNet_R-101-FPN_1x	37.7	38.5	0.123	model
RetinaNet_X-101-32x8d-FPN_1x	39.5	39.8	0.200	model
RetinaNet_R-50-FPN_P5_1x	35.7	36.2	0.097	model
RetinaNet_R-101-FPN_P5_1x	37.7	38.5	0.121	model

Add _C.TEST.DETECTIONS_PER_IMG = 100.
After using DETECTIONS_PER_IMG, the mAP drops 0.1.

Not Implemented parts.

• Class specific bbox prediction.
• Softmax Focal Loss

---

Updated 02/02/2018
Identify the reason why this branch gets higher AP.

Branch	Accuracy	Difference
This	36.3/55.2/38.9/ 19.7/39.9/49.0	BoxCoder(10, 10, 5, 5), add *4 in classification loss normalization
retinanet-detectron	35.6/55.8/37.7/ 19.6/39.3/48.2	BoxCoder(1, 1, 1, 1)

---

Updated 01/30/2018
After updating PyTorch to v1.0.0, the inference time reduced around 15~20%.
Update the inference time in the table.

---

Updated 01/26/2018
Add RetinaNet_X-101-32x8d-FPN_1x model.
AP : 39.8
Inferece time : 0.200 second.

---

Updated 01/25/2018
In my first version, I accidentally used P5 to generate P6 instead of C5 which was used in Detectron and paper.
The following table compares the performances in these two settings.

Model	C5	P5
RetinaNet_R-50-FPN_1x	36.3/55.2/38.9/19.7/39.9/49.0	36.2/55.1/38.7/19.7/39.5/48.6
RetinaNet_R-101-FPN_1x	38.5/57.6/41.0/20.8/42.3/51.7	38.5/57.9/41.3/21.0/42.8/51.3

---

Updated 01/23/2018

Train the model without "divide by 4" in the regression loss.
Performance:

Model	AP	AP50	AP75	APs	APm	APl
RetinaNet_R-50-FPN_1x	29.6	45.0	31.5	13.9	31.4	41.2

---

Updated 11/20/2018

The matching part is slightly different from the Detectron version.
In Detectron matching, anchors with IOU >= 0.5 are considered as positive examples, and anchors with IOU <=0.4 are negative examples. Then for the low qualities matches(best prediction for each gt), Detectron only uses the low-quality examples >= 0.4.

P.S.In Detectron, there are some cases occur in both fg_inds and bg_inds. Although in Line230, Detectron removes all the low-qualities positive examples < 0.4. I think the in Line231, num_fg calculation is not correct.

I also test the threshold used for low-qualities positive examples from 0.5 to 0.0.

threshold	AP	AP50	AP75	P.S.
0.5	35.5	53.7	38.1
0.4	36.0	54.1	38.6	Detectron version.
0.3	36.1	54.5	38.9
0.2	36.1	54.5	38.7
0.0	36.2	55.0	38.7	This branch.

[facebook-github-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need the corporate CLA signed.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

[fmassa]

This is really awesome, thanks a lot for the PR!

I'll have a closer look at it next week, let us know the result of the training!

[chengyangfu]

Finishing the training.
RetinaNet with X_101_32x8d backbone model costs too much time for training now. Due to the CVPR submission deadline is coming, our lab does not have extra machines for training this one. If anyone can train this one, I will highly appreciate it!

[fmassa]

No worries about X_101_32x8d training, we can do it on our side.

[fmassa]

Once again thanks a lot for this awesome PR!

This is not a complete review yet.

One question I have is that I think we might want to move _C.RETINANET into _C.MODEL.RETINANET, but let's wait until @rbgirshick comment on that.

[rbgirshick]

@chengyangfu nice work! Do you know what implementation differences might have caused the improvement in box AP relative to the Detectron implementation?

I'm also curious if you need to use a C++ implementation of sigmoid focal loss or if you can simply use a Python implementation using torch.nn.functional? Ideally it could be simplified to the Python version.

[chengyangfu]

@rbgirshick
For the C++ implementation of sigmoid focal loss, I have tested this in my another project. Python version needs to discard ignored examples first and then calculate focal loss, but C++/CUDA version can combine focal loss and selection. The C++/CUDA version definitely has lower memory footprints and runs faster. The critical part is the selection. If I called labels >=0 and use it to get the positive and negative examples, the performance will drop quickly. Due to a large number of positive and negative examples in the training, I think that's reasonable.

The following is the python version of Focal Loss I tested.

   def forward(self, inputs, targets):
       N = inputs.size(0)
       C = inputs.size(1)
       class_mask = inputs.new_zeros((N, C))
       ids = targets.view(-1, 1)
       class_mask.scatter_(1, ids, 1.)

       class_mask = class_mask[:, 1:]
       inputs = inputs[:, 1:]

       P = torch.sigmoid(inputs)
       PC = P*class_mask + (1-P)*(1-class_mask)
       alpha = self.alpha * class_mask + (1 - self.alpha) * (1 - class_mask)
       focal_weight = alpha * (1 - PC).pow(self.gamma)
       loss = F.binary_cross_entropy_with_logits(inputs, class_mask,
                                                     focal_weight)
       return loss

[zimenglan-sysu-512]

hi @laibe
i try to run your yaml files, it encounter OOM (IMS_PER_BATCH=2)

[fmassa]

Hi @chengyangfu ,

Thanks for the benchmark!

I believe this is a consequence of the operations not being fused in PyTorch 1.0.0. I think there have been some improvements recently that made it better, but I'd need to check.

cc @ailzhang for the performance timings and memory

Let's keep the CUDA implementation for now then, and dispatch to the Python implementation if we the tensor is on CPU, how does that sound?

Then this will be ready to be merged.

Once again, thanks a lot for this awesome contribution!

[laibe]

@zimenglan-sysu-512 what's your training setup? I was using:
1x GeForce GTX 1080 Ti
CUDA runtime version: 9.0.176
with max mem during training of 5132

[zimenglan-sysu-512]

thanks @laibe
i make a mistake resulting in OOM.
btw, do u use CUDA implementation or Python implementation?

after using cuda version, retinanet_MobileNetV2-96-FPN_1x needs 5146 mem, while retinanet_MobileNetV2-FPN_1x.pth needs 5486 mem.

[ailzhang]

The numbers might make sense given the current fusion logic in jit, @waochaol @zou3519 could you also help check on the JIT numbers? Thanks!

[chengyangfu]

@fmassa
It sounds good to me. I just updated the maskrcnn_benchmark/layers/sigmoid_focal_loss.py according to the suggestion.

[fmassa]

Great, thanks!

[fmassa]

That's an awesome contribution @chengyangfu , thanks a lot for all your effort!

[buaaMars]

@chengyangfu
Hi,
Great work!
I have a question on rpn/retinanet/inference.py(77)
Why do you reshape the box_regression when it has been permute_and_flatten just in the last line? According to rpn/utils.py(13), (N, -1, 4) is exactly the shape of box_regression. Do you reshape it again in order to handle with some spacial cases?

thks a lot

[as1392]

Ah... I finally realized why model zoo does not have these trained weights... Removing OUT_CHANNELS: 256 from backbone destroyed trained networks? I hope someone update/convert these weights :(

Edit : OK, never mind this comment. It was just giving scores lower than 0.7(on the whole image). Try predeictions on COCO_val2014_000000355257.jpg.

[dedoogong]

could you please support Sigmoid Focal Loss cuda implementation to run on FP16?

Thank you

[simaiden]

Can I use this model to train my custom dataset as in #521 ?

[chenjoya]

supplement performance of retinanet_r101fpn_2x on COCO minival:
AP, AP50, AP75, APs, APm, APl
0.3878, 0.5811, 0.4132, 0.2081, 0.4256, 0.5183

[adizhol]

Hi @chengyangfu :)
Why is the focal loss (the sum of the losses) divided by the number of positive labels plus number of labels (N = len(labels))?

retinanet_cls_loss = self.box_cls_loss_func(
            box_cls,
            labels
        ) / (pos_inds.numel() + N)