The original data is only 1000, which is not enough to generate 5 H5 files

[gqsmmz]

When running python generate_pre_data.py, I found that the training data bc_train_check. json only had 10000 pieces of data, and the validation dataset only had 2953 pieces of data. It is not enough to generate 5 h5 files from [5999,11999,17999,23999,25616] like the following equation.

Is this because only a portion of the original data was uploaded?

[whcpumpkin]

Hi,
Did you download the raw_trajectory_dataset.zip in the Materials Download in OneDrive?
unzip this zip file, and run the following code:

import json
with open('bc_train_check.json', 'r') as f:
    data = json.load(f)
print("total number of bc_train_check.json: ", len(data))

the output is total number of bc_train_check.json: 25617

I am not sure whether the zip file is different bewteen in onedrive and googledrive even though they are the same in size (6.2GB)

[whcpumpkin]

I also recommend downloading the processed data directly in the onedrive

[gqsmmz]

Thank you for your reply, there is no problem with this data!

[AlooTikkiii]

I hope ur problem is with the code file : generate_pre_data line : 81. The issue is with min(len(data), args.end) where args.end is 10k. Hope that solves ur issue