Training Problem

[cvrookie11]

Hi, sorry to interrupt!

I used the following code for training with the same data.

python main.py --gpu=0 --task=cable-shape --agent=transporter --crop_bef_q=0 --num_demos=1000

When num_demos is set to 0,1,10,100, the training completes without a problem, but when num_demos is set to 1000, each training will force a kill at 1000 + iterations.

My GPU is an RTX 4090 with 24GB memory. I looked into whether GPU memory was causing the problem and tweaked --gpu_mem_limit, but it still didn't work. I didn't make any other changes to the provided main.py. Do we need to make some parameter changes to main.py when num_demos=1000, or what else is causing this?

Looking forward to your reply!

[cvrookie11]

Furthermore, I tried to set num_demos to 500 and still trained successfully, but increasing it to 800 resulted in the process being killed after more than 1000 iterations.

[DanielTakeshi]

"each training will force a kill at 1000 + iterations."

It helps if you share the exact error message.