No such file or directory: '/nfs-gpu/xlstm'

[KarolinaPondel]

Hi,
I am trying to run xlstm to nlp on my gpu. Unfortunately when running data preparation:
python scripts/data_processing/hf_to_arrayrecord.py --hf_path=DKYoon/SlimPajama-6B
I get an error: No such file or directory: '/nfs-gpu/xlstm'
What does this mean? I have no such nfs-gpu directory anywhere. Is it some specific method of accessing gpu?
I also have more errors like that:
No such file or directory: '/nfs-gpu/xlstm/data'
No such file or directory: '/nfs-gpu/xlstm/data/array_records'
No such file or directory: '/nfs-gpu/xlstm/data/array_records/mlfoundations_dclm-baseline-1.0-parquet-split'
And finally: PermissionError: [Errno 13] Permission denied: '/nfs-gpu'

[discimus-scientia]

Hi @KarolinaPondel thanks for noticing! It is complaining that you don't have the folder /nfs-gpu
These paths should not be in the code, we will fix that.
For now, you can replace any occurence of nfs-gpu with your local path.

[KarolinaPondel]

Thanks! I changed it and now everything works. You have to do it in all scripts with data_processing, everywhere this path is hardcoded. Maybe consider adding a variable in the parser to pass the path to the output directory? It would be convenient :)