The model lung_nodule_ct_detection did not change after training

[PathSally]

Hello👋
I used the Lung nodule ct detection model in 3D slicer, and firstly obtained some prediction results by using the model. I trained it in the 3d slicer after modifying the results of multiple images.
Unfortunately, I haven't noticed any changes in the improved model. I wonder what can be done about it? Thanks!

[yiheng-wang-nv]

Hi @PathSally , as for the models, have you downloaded the related dataset, and check the val metric before and after you changed your model?
Could you please provide more details about your experiments (steps, hyper-parameter settings, ...) thus we can help to debug.

[PathSally]

Hi, the dataset I use is the local image dataset, which has been converted to nii.gz.
Just now I tried again and found the error. As follows:

[2023-03-06 11:22:33,804] [2719] [MainThread] [INFO] (ignite.engine.engine.DetectionTrainer:138) - Restored all variables from /root/apps/monaibundle/model/lung_nodule_ct_detection/models/model.pt
[2023-03-06 11:22:35,534] [2719] [MainThread] [ERROR] (ignite.engine.engine.DetectionTrainer:1086) - Current run is terminating due to exception: Caught RuntimeError in DataLoader worker process 0.
...
File "/root/miniconda3/lib/python3.8/site-packages/monai/transforms/transform.py", line 129, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f71c8238d00>
[2023-03-06 11:22:36,803] [591] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 1
😥I was confused by these errors.

[tangy5]

Hi @PathSally , this error mean the length of dataloader is 0. Could you double check the number of data samples labeled in the labels/final directory, you could also debug in the train.json file. Thanks.

[PathSally]

Thank you for your answer. This problem was solved after I submitted label for all the data in the datasets folder.

However, a new problem appeared, and I tried to search for the relevant solution, but got no answer. I also tried using the data set provided in the tutorial and encountered these errors while training in the 3d slicer.
The following errors occurred several times in the retraining:
RuntimeError: applying transform <monai.apps.detection.transforms.dictionary.ConvertBoxToStandardModed object at 0x7f454c08b3a0>
IndexError: tuple index out of range
RuntimeError: applying transform <monai.apps.detection.transforms.dictionary.ConvertBoxToStandardModed object at 0x7f454c08b3a0>
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f454c099f10>