CUBLAS_STATUS_EXECUTION_FAILED and .h5 file not found

[joeychsu]

Hello! I'm interested in this research and would like to try running it myself.
I have referred to a solution for this issue, but it seems to be slightly different.
After I finished json2pc.py and get all training / testing point clouds. (except INVALID_IDS)
When I run pc2cad.py, I got the following error message:
unable to open file: name = 'proj_log/pretrained/results/all_zs_ckpt1000.h5'
Where I can get this file?

In the other hand, When I run test.py , I got the following error message:
python test.py --exp_name pretrained --mode rec --ckpt 1000 -g 0
----Experiment Configuration-----
proj_dir proj_log
data_root data
exp_name pretrained
gpu_ids 0
batch_size 512
num_workers 8
nr_epochs 1000
lr 0.001
grad_clip 1.0
warmup_step 2000
cont False
ckpt 1000
vis False
save_frequency 500
val_frequency 10
vis_frequency 2000
augment False
mode rec
outputs None
z_path None
Loading checkpoint from proj_log/pretrained/model/ckpt_epoch1000.pth ...
Total number of test data: 16
0%| | 0/16 [00:00<?, ?it/s]Traceback (most recent call last):
File "test.py", line 130, in
main()
File "test.py", line 18, in main
reconstruct(cfg)
File "test.py", line 52, in reconstruct
outputs, _ = tr_agent.forward(data)
File "/home/joehsu/DeepCAD/trainer/trainerAE.py", line 27, in forward
outputs = self.net(commands, args)
File "/home/joehsu/miniconda3/envs/deepcad/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/joehsu/DeepCAD/model/autoencoder.py", line 154, in forward
z = self.encoder(commands_enc_, args_enc_)
File "/home/joehsu/miniconda3/envs/deepcad/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/joehsu/DeepCAD/model/autoencoder.py", line 74, in forward
src = self.embedding(commands, args, group_mask)
File "/home/joehsu/miniconda3/envs/deepcad/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/joehsu/DeepCAD/model/autoencoder.py", line 32, in forward
self.embed_fcn(self.arg_embed((args + 1).long()).view(S, N, -1)) # shift due to -1 PAD_VAL
File "/home/joehsu/miniconda3/envs/deepcad/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/joehsu/miniconda3/envs/deepcad/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/home/joehsu/miniconda3/envs/deepcad/lib/python3.7/site-packages/torch/nn/functional.py", line 1612, in linear
output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

other info :

NVIDIA GeForce RTX 3080 Ti
CUDA Version: 11.7
Ubuntu 20.04.6 LTS
Pytorch = 1.5.1
Python = 3.7