Alignment problems with larger files (sambamba-sort: Failed to remove file)

[viktorlj]

Hi!

I have been using bcbio for a while with success but has since about a month back encountered a problem in BWA alignment when it comes to larger files (exome and wgs). Alignment starts off ok and seems to perform a full alignment but in the final step where sambamba should kick in and sort the files it crashes with the following output:

sambamba-sort: Failed to remove file /pica/v6/b2013224_nobackup/private/MDS_KI_20150309/MDS_Exome/work/align/PV1367-Tcells/tx/tmpWEdyFW/2_2015-03-11_MDS_Exome-sort-sorttmp-full/sambamba-pid1594-bugk/.nfs0000000029faf37f00000036: Device or resource busy
' returned non-zero exit status 1

What I find strange is that it only seem to affect larger files. When I test run some targeted sequencing files and a test file with 500 000 reads there is no problem. I have previously been able to align exomes without troubles so this is something new. The problem to some extent resembles the one discussed here #747 but with the difference that an upgrade with the development flag does not resolve the problem. I've tried to downgrade sambamba to 5.0 but this does not change anything.

I have uploaded the log files of a test run here: https://drive.google.com/folderview?id=0B-L4wCdmSz0CMk9lNDdKS3pWczA&usp=sharing

The first sample is a test sample with 500K reads and as you can see there is no trouble there. The second sample is an exome run and that one crashes in the end. All temp files and folders are whiped after the crash so it's hard to tell what the structure looks like.

Sorry if this is something obvious or non bcbio-related, but I'm a bit stuck right now and would appreciate any pointers to what could be done. Please let me know if any other info would be helpful.

[chapmanb]

Thanks for the report and sorry about the problems. What's happening here is that the NFS share filesystem is unable to keep up with the additional load imposed by the larger files. When sambamba tries to delete a temporary directory it fails because NFS is still struggling to get it up to date and still has a lock on some of the files.

@lomereiter, would it be possible for sambamba to ignore/re-try errors when trying to clean up the temporary directory to avoid this situation?

@viktorlj If your machine has local scratch space, you can avoid this issue by telling bcbio to use it instead of the shared NFS filesystem:

https://bcbio-nextgen.readthedocs.org/en/latest/contents/configuration.html#temporary-directory

This is the best practical way to avoid the issue right now. Hope this helps.

[viktorlj]

Thank you so much! I set the tmp to the node scratch surface and now it runs perfect.