FAQ: LoFreq gives error "OSError: [Errno 7] Argument list too long: lofreq

[DennisSchmitz]

I will update this post if more information becomes available. See also related issue #102.

On certain off-site servers we see the error shown below. Even with datasets that have previously, reproducibly/reliably, finished on RHEL, CentOS, Ubuntu and WSL1(Ubuntu) systems. This leads us to believe this is a client-sided problem. Most likely due to a limitation in the number of simultaneous open files and/or a stack size limit (see ulimit -Sa and ulimit -Ha). LoFreq seems to send a very big command that possibly exceeds the stack size limit. Also, it uses many temporary files, which might trigger the simultaneous open file limit. Locally we have the limits shown below and on servers that give the errors we generally see lower limits:

open files          4096
stack size          unlimited

In rule SNP_calling:

INFO [2020-05-03 17:05:02,738]: Adding 49278 commands to mp-pool
Traceback (most recent call last):
  File "/mnt/StudentFiles/2019-20/Project03/20200501_Jovian_DS/.snakemake/conda/e8de1337/bin/lofreq2_call_pparallel.py", line 746, in <module>
    main()
  File "/mnt/StudentFiles/2019-20/Project03/20200501_Jovian_DS/.snakemake/conda/e8de1337/bin/lofreq2_call_pparallel.py", line 669, in main
    "##source=%s" % ' '.join(sys.argv))
  File "/mnt/StudentFiles/2019-20/Project03/20200501_Jovian_DS/.snakemake/conda/e8de1337/bin/lofreq2_call_pparallel.py", line 174, in concat_vcf_files
    subprocess.check_call(cmd)
  File "/mnt/StudentFiles/2019-20/Project03/20200501_Jovian_DS/.snakemake/conda/e8de1337/lib/python3.6/subprocess.py", line 306, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/mnt/StudentFiles/2019-20/Project03/20200501_Jovian_DS/.snakemake/conda/e8de1337/lib/python3.6/subprocess.py", line 287, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/mnt/StudentFiles/2019-20/Project03/20200501_Jovian_DS/.snakemake/conda/e8de1337/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/mnt/StudentFiles/2019-20/Project03/20200501_Jovian_DS/.snakemake/conda/e8de1337/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: 'lofreq'

[DennisSchmitz]

Update. Indeed, after increasing these limits it finishes without error. This is a client-sided problem and thus out-of-scope of Jovian. If you encounter this error, please contact your sys-admin and ask him/her to increase these limits.

Leaving this issue open for clarity.