Discrepancy in the validation IOU values for Driveable Area segmentation

[linlion0311]

I have implemented the author's code, but I noticed a discrepancy in the validation IOU values for road segmentation (In Paper: Driveable Area = 74.3, My implementation = 68.7). However, the vehicle category scores are consistent with the scores mentioned in the paper. I would like to ask whether I should consider changing the training labels or how I should proceed with the training. Thank you!

[flymin]

I cannot reproduce the results reported in the paper, either. Even with the released checkpoint, the result from validation is only 72.21. Any hint for this?

[linlion0311]

I cannot reproduce the results reported in the paper, either. Even with the released checkpoint, the result from validation is only 72.21. Any hint for this?

Are you using released checkpoint from Github to do validation? Or you using your model

[flymin]

Are you using released checkpoint from Github to do validation? Or you using your model

I use the checkpoint from this repo for validation, and get 72.21 for the driveable area.

[linlion0311]

Are you using released checkpoint from Github to do validation? Or you using your model

I use the checkpoint from this repo for validation, and get 72.21 for the driveable area.

Please note your dataset splitting method. I noticed that the author used 'road segment' as the label for drivable areas. After making changes, I achieved an IOU of 73.3.

By the way, could you please share how you use checkpoints to validate the model? I am not very familiar with Python programming. If possible, could you kindly send the complete implementation process to linlion0311@gmail.com? Thank you very much!

[flymin]

Hi @linlion0311,

I think in this line, the author defined driveable area as a combination of road segment and lane. Why do you change it to 'road segment'? Is is correct for the settings in the paper?

Sorry I cannot share my source code, you can call trainer.validate(model_module, val_loader) similar to this line

Update: with the released ckpt and cvt_labels_nuscenes_v2, I got 73.69 for road. Even if it is better, it still cannot catch up with 74.3 reported in the paper.

[linlion0311]

It's ok, I'll keep studying it.

Regarding the label segmentation part, I tested using 'lane' + 'road segment', and it converged faster.
Perhaps this is the reason why the author didn't use 'lane' + 'drivable area' (they are very similar). have a good day!

[linlion0311]

Hi @flymin

Can I ask if you are implementing this repo in a Windows or Ubuntu(Linux) environment?

because I can't implement it in Windows.

[flymin]

Hi @flymin

Can I ask if you are implementing this repo in a Windows or Ubuntu(Linux) environment?

because I can't implement it in Windows.

I use Linux. Sorry I cannot help you with Windows.

[1146065545]

Hi @flymin

Can I ask if you are implementing this repo in a Windows or Ubuntu(Linux) environment?

because I can't implement it in Windows.

Hello, could you please tell me how the code provided by the author can test the trained model and how to visualize it? I've been doing this lately as well, thank you very much!

[flymin]

Please check the dialogue above. I think you’d better check the doc of pytorch-lightning for testing. And for visualization, I remember the author has provided a notebook.

[1146065545]

Please check the dialogue above. I think you’d better check the doc of pytorch-lightning for testing. And for visualization, I remember the author has provided a notebook.

Sorry, can you go into more detail, I mean about testing the trained model, does the author provide specific .py files

[flymin]

I don’t think so.

[1146065545]

I don’t think so.

╥﹏╥

[SunJ1025]

@flymin hi flymin, I want to ask which .yaml did you use to get "73.69 for road." I only got about 47/48 when I test with the commond python3 scripts/train.py +experiment=cvt_nuscenes_road data.dataset_dir=/data1/dataset/nuscenes data.labels_dir=/data1/dataset/nuscenes/cvt_labels_nuscenes_v2

I test with the released cvt_nuscenes_road_75k.ckpt and cvt_labels_nuscenes_v2.

[KimJunHan]

@flymin hi flymin, I want to ask which .yaml did you use to get "73.69 for road." I only got about 47/48 when I test with the commond python3 scripts/train.py +experiment=cvt_nuscenes_road data.dataset_dir=/data1/dataset/nuscenes data.labels_dir=/data1/dataset/nuscenes/cvt_labels_nuscenes_v2

I test with the released cvt_nuscenes_road_75k.ckpt and cvt_labels_nuscenes_v2.

How did you find IOU value like picture ???