The results are inconsistent with those in the paper

[TriBall3]

Hi!

I used your code to train the Cable-Shape model with Transporter. In the final test, I found that the success rate was quite different from that in the paper. When I learned 1000 demos, the success rate in the paper was 86.5%, but the highest success rate was only 70%.I used the load.py file in the test and tested 100 cases.

Why are my results so different from yours?

Did you use the default 20 cases in your code when testing, or did you choose 100?

The following results were tested with a model of 25,000, 30,000, 35,000 and 40,000 steps respectively.

[DanielTakeshi]

Did you use the same data that I used? If so, then we should dig deeper into this.

[DanielTakeshi]

I think we used 20 cases, since in the appendix + main part of the paper I report that we used 20 held-out random seeds.

[TriBall3]

Instead of using the expert demo data you provided, I generated 1000 demo data using the following code:
python main.py --gpu=0 --agent=dummy --hz=240 --task=cable-shape

After the data was generated, I used the following code for training, training 20000 steps, step length is 2000, a total of 10 models, training for three times.
python main.py --gpu=0 --task=cable-shape --agent=transporter --crop_bef_q=0 --num_demos=1000

Test the trained model with the following code：

HZ=240
ROTS=1
AGENT=transporter   # transporter, gt_state, gt_state_2_step
TASK=cable-shape    # cable-shape, cable-ring, cable-ring-notarget

for tr in 0 1 2; do
    python load.py --gpu=0 --agent=${AGENT} --hz=${HZ} --task=${TASK} --num_rots_inf=${ROTS} --train_run=${tr} --num_demos=1000
done

The figure below is the test result of three models with 20,000 steps, but the success rate has not reached the result in the paper. 20 cases were used in the test.

Is it possible that I did not use your expert presentation data, leading to the deviation of the results?
I do not know whether my experimental steps are correct, please correct,Thank you very much!

[TriBall3]

Can you share the models in your checkpoint folder? Thanks!

[DanielTakeshi]

I don't think I have the models I used for this paper unfortunately. :( I had them saved at some point, however GCTN take up a lot of RAM and the experiments involved running multiple runs of BC for {1,10,100,1000} demos.

[TriBall3]

I don't think I have the models I used for this paper unfortunately. :( I had them saved at some point, however GCTN take up a lot of RAM and the experiments involved running multiple runs of BC for {1,10,100,1000} demos.

Thank you for your reply!However, I still don't know why my test results are different from yours. I have trained the model and tested it with the demonstration data provided by you, and my results are as follows.

This is my training test step, where did I go wrong?#11 (comment)

Thank you very much for your time!

[TriBall3]

Hi!
I hope you can take time out of your busy schedule to answer my questions, I will be very grateful!
Thank you very much!

[DanielTakeshi]

Apologies @TriBall3 I'm consumed with the CoRL 2022 deadline, happy to take a look at this afterwards.

[TriBall3]

Hi!
Have you finished your CoRL paper? If you are finished, please take a little time to answer my above questions. I will be grateful !
Thank you very much !

[DanielTakeshi]

I was able to re-run a lot of the data recently after a bug fix
#15
You should in general get better results after incorporating this fix.