Evaluate on Nuscenes with Fcos3d but get low mAP

[xingyoujun]

HI,I try to evaluate on Nuscenes with pre-trained Fcos3d model but find mAP=23% compared with 29% in your paper.Other metrics looks right.My results with nms_thr=0.8 and score_thr=0.05 are as follows:

I notice some issues mentioned that lower nms_thr or higher score_thr may remove redundant preds. So I adjustment nms_thr=0.5 and score_thr=0.07,my results show that mAP is still around 23%:

Finally,I do visualization and find this situation.
Even I set nms_thr=0.01 and score_thr=0.08,there are redundant preds which should be remove by NMS.

When I ran into the code,I find this line doing NMS operation.It looks like change the order of boxes howerver not change the order of score.So the return from nms_rotated() is in the origin order same as score but mismatches with boxes order.So I tried drop this line and find results much better with mAP=28%. With nms_thr=0.1 and score_thr=0.1

with visualization

It seems right now.However traffic_cone on the right still has too much pred boxes.As shown in the image,it has three very close pred boxes which seems should be filtered by NMS.I wonder if it is caused by the different depths,or other reasons?This is my second question.

The last one is in another issue.According to his description,his visualization result has too many 3D boxes but the mAP=29% is right.I thought the redundant 3d box made my mAP low.His issue really makes me confused.

By the way, my eval command is as follow.With 4 3090.

tools/dist_test.sh /mmdetection3d/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py mmdetection3d/model_weighs/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_20210715_235813-4bed5239.pth 4 --eval bbox

My running environment is
PyTorch: 1.8.1+cu111 OpenCV: 4.5.4 MMCV: 1.4.8 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 11.1 MMDetection: 2.24.1 MMSegmentation: 0.24.1 MMDetection3D: 1.0.0rc2+76e351a spconv2.0: False

I look forward to your early reply! Thank you very much.

[YinengXiong]

Hello, do you mean commenting the line
boxes = boxes[order].contiguous() in bbox3d_nms.py can solve the problem of producing redundent box in FCOS3D

[xingyoujun]

Hello,in my test,I think nms_bev() in bbox3d_nms.py didn't work right because even i change nms_thr=0.0001, there are still redundent box.And mAP is still around 23%.

After commenting the line boxes = boxes[order].contiguous(),the most box can be removed correctly by NMS,and nms_thr=0.1 and score_thr=0.1,I got result above with mAP=28%.

By the way, I only find nms_bev() in the latest released version.In the earlier version, there won't have this order problem.

[filaPro]

Hi @xingyoujun ,

Sorry for this. It should be fixed in #1479. Can you please check if it helps?

[xingyoujun]

@filaPro Thanks for your reply.It solves my problem well.Thank you!

[LMerCy]

@xingyoujun what's your version of mmdet3d and mmcv? I still have the problem after commenting boxes = boxes[order].contiguous() in bbox3d_nms.py