segmentation fault during consensus

[bernardo1963]

Hi,
I am using verkko 2.1 (bioconda verkko bioconda 2.1) in a local machine with 208 threads and 2 TB RAM, running CentOS7. I am assembling the Y chromosome of HG002 using subsets of the HiFi and ONT UL reads, and I am repeatedly getting a segmentation fault error during the consensus step. The HiFi and ONT reads were obtained as follows. I used minimap2 and seqtk to get the Y-linked reads from HiFi and ONT reads, and assembled them with verkko. Several variations worked well, but one paticular read dataset (HiFi + ONT) is always causing a fatal error (I uploaded these reads to https://drive.google.com/drive/folders/1kp4zPwQ27rNNzvXeM-3MdUT5slNXiLUH?usp=sharing). 2024-06-20T113029.669398.snakemake.log The command line I am using is:

conda activate verkko
nohup verkko -d Y_asm2  --local-memory 1000   --hifi HiFi35x_Yreads_clean.fastq.gz  --nano ONT60x_Yreads_clean.fastq.gz --threads 100 >  nohup_Y_asm2.txt &

I attached the snakemake log file.
The error message follows:

 Consensus did not finish successfully, exit code \$?.
69 of 72 steps (96%) done
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-06-17T214529.496147.snakemake.log

Digging a bit more, I found no error message in ./7-consensus/buildPackages.err , but there is a detailed report in 7-consensus/packages/part005.err , as follows in the end of this message.
I obtained the reads twice (using seqtk)), with the same error result. So now I am stuck. Any suggestions?
thanks,
Bernardo

# -- Opening output FASTA file '../7-consensus/packages/part005.fasta.WORKING'.
# --
# -- Computing consensus for b=0 to e=4294967295 with errorRate 0.0500 (max 0.2000) and minimum overlap 3000
# --
# alignEdLib()-- WARNING: tigbgn 533523 > tigend 531512 - tiglen 531512 utgpos 533773-549246 padding 250
# alignEdLib()-- WARNING: updated tigbgn 0 > tigend 549246 - tiglen 531512 utgpos 533773-549246 padding 250
# alignEdLib()-- WARNING: tigbgn 534370 > tigend 531512 - tiglen 531512 utgpos 534620-554990 padding 250
# alignEdLib()-- WARNING: updated tigbgn 0 > tigend 554990 - tiglen 531512 utgpos 534620-554990 padding 250
# alignEdLib()-- WARNING: tigbgn 536266 > tigend 531512 - tiglen 531512 utgpos 536516-549917 padding 250
# alignEdLib()-- WARNING: updated tigbgn 0 > tigend 549917 - tiglen 531512 utgpos 536516-549917 padding 250
# alignEdLib()-- WARNING: tigbgn 536530 > tigend 531512 - tiglen 531512 utgpos 536780-548120 padding 250
# alignEdLib()-- WARNING: updated tigbgn 0 > tigend 548120 - tiglen 531512 utgpos 536780-548120 padding 250
# ERROR:  alignment from 535366 to 546625, but tiglen is only 531512
# utgcns: utgcns/unitigConsensus.C:869: bool alignEdLib(dagAlignment&, tgPosition&, char*, uint32, char*, uint32, double, bool): Assertion `aln.end <= tiglen' failed.
                           # ----------CONTAINED READS----------  -DOVETAIL  READS-
  # tigID    length   reads      used coverage  ignored coverage      used coverage
# ------- --------- -------  -------- -------- -------- --------  -------- --------
      # 6     14740       4         2    1.19x        0    0.00x         2    1.68x
     # 11   1572918    2107      1419    8.93x        0    0.00x       688    5.87x
     # 18    469157     535       345    7.86x        0    0.00x       190    5.75x
     # 30    453951     181       175    8.88x        0    0.00x         6    2.39x

# Failed with 'Aborted'; backtrace (libbacktrace):

# Failed with 'Segmentation fault'; backtrace (libbacktrace):
# ./packages/part005.sh: line 18: 293115 Segmentation fault      (core dumped) /home6/tools/miniconda3/envs/verkko/lib/verkko/bin/utgcns -threads 8 -import ../7-consensus/packages/part005.cnspack -A ../7-consensus/packages/part005.fasta.WORKING -C 2 -norealign -maxcoverage 50 -e 0.05 -em 0.20 -EM 0 -l 3000 -edlib

# Consensus did not finish successfully, exit code 0.

# Files in current directory:
# total 8896
# -rwxr-xr-x 1 bernardo tools    1215 Jun 20 10:08 extractONT.sh
# -rw-r--r-- 1 bernardo tools      48 Jun 20 10:08 extractONT.err
# -rw-r--r-- 1 bernardo tools    3848 Jun 20 10:08 ont_subset.id
# -rw-r--r-- 1 bernardo tools 4853742 Jun 20 10:08 ont_subset.fasta.gz
# -rw-r--r-- 1 bernardo tools      86 Jun 20 10:08 ont_subset.extract
# -rwxr-xr-x 1 bernardo tools     447 Jun 20 10:08 buildPackages.sh
# -rw-r--r-- 1 bernardo tools 4199297 Jun 20 10:08 packages.readName_to_ID.map
# -rw-r--r-- 1 bernardo tools   18024 Jun 20 10:09 buildPackages.err
# -rw-r--r-- 1 bernardo tools     870 Jun 20 10:09 packages.tigName_to_ID.map
# -rw-r--r-- 1 bernardo tools    2855 Jun 20 10:09 packages.report
# -rw-r--r-- 1 bernardo tools       0 Jun 20 10:09 packages.finished
# drwxr-xr-x 2 bernardo tools    4096 Jun 20 10:11 packages

# Files in packages/:
# total 311764
# -rw-r--r-- 1 bernardo tools 159536904 Jun 20 10:09 part001.cnspack
# -rw-r--r-- 1 bernardo tools  12873952 Jun 20 10:09 part005.cnspack
# -rw-r--r-- 1 bernardo tools  12977304 Jun 20 10:09 part004.cnspack
# -rw-r--r-- 1 bernardo tools  13103980 Jun 20 10:09 part003.cnspack
# -rw-r--r-- 1 bernardo tools  61223936 Jun 20 10:09 part002.cnspack
# -rw-r--r-- 1 bernardo tools  11476064 Jun 20 10:09 part007.cnspack
# -rw-r--r-- 1 bernardo tools  11586852 Jun 20 10:09 part006.cnspack
# -rw-r--r-- 1 bernardo tools   6079000 Jun 20 10:09 part008.cnspack
# -rwxr-xr-x 1 bernardo tools       605 Jun 20 10:09 part006.sh
# -rwxr-xr-x 1 bernardo tools       605 Jun 20 10:09 part007.sh
# -rwxr-xr-x 1 bernardo tools       605 Jun 20 10:09 part005.sh
# -rwxr-xr-x 1 bernardo tools       605 Jun 20 10:09 part004.sh
# -rwxr-xr-x 1 bernardo tools       605 Jun 20 10:09 part003.sh
# -rwxr-xr-x 1 bernardo tools       605 Jun 20 10:09 part001.sh
# -rw-r--r-- 1 bernardo tools         0 Jun 20 10:09 part001.fasta.WORKING
# -rw-r--r-- 1 bernardo tools       191 Jun 20 10:09 part001.err
# -rwxr-xr-x 1 bernardo tools       605 Jun 20 10:09 part002.sh
# -rwxr-xr-x 1 bernardo tools       605 Jun 20 10:09 part008.sh
# -rw-r--r-- 1 bernardo tools       191 Jun 20 10:09 part008.err
# -rw-r--r-- 1 bernardo tools   1032192 Jun 20 10:09 part008.fasta.WORKING
# -rw-r--r-- 1 bernardo tools       771 Jun 20 10:09 part006.err
# -rw-r--r-- 1 bernardo tools   2669120 Jun 20 10:09 part006.fasta
# -rw-r--r-- 1 bernardo tools       853 Jun 20 10:09 part007.err
# -rw-r--r-- 1 bernardo tools   2727324 Jun 20 10:09 part007.fasta
# -rw-r--r-- 1 bernardo tools       525 Jun 20 10:09 part003.err
# -rw-r--r-- 1 bernardo tools   3338554 Jun 20 10:09 part003.fasta
# -rw-r--r-- 1 bernardo tools       525 Jun 20 10:09 part004.err
# -rw-r--r-- 1 bernardo tools   3051813 Jun 20 10:09 part004.fasta
# -rw-r--r-- 1 bernardo tools       525 Jun 20 10:11 part002.err
# -rw-r--r-- 1 bernardo tools  14652343 Jun 20 10:11 part002.fasta
# -rw-r--r-- 1 bernardo tools   2817683 Jun 20 10:12 part005.fasta.WORKING
# -rw-r--r-- 1 bernardo tools      3174 Jun 20 10:12 part005.err

[skoren]

I was able to run this locally with v2.1 without any consensus errors/warnings. I suspect this is the same as issue #247 and is fixed by this commit: marbl/canu@c61ebbb. In that case, you'd need to build canu locally and replace the utgcns binary in your conda installation of verkko. Before you do that though, I'd like to verify if that's the case, can you share the cnspack for one of the failing partitions so I can run it locally?

[bernardo1963]

Dear Serge,
Thank you for your help. I uploaded to the same GoogleDrive folder all part005 files (including part005.cnspack ). part005 is the one that produced the fatal error.

[skoren]

Thanks, I confirmed the crash and that the reference commit above fixes the issue. It will be in the next verkko release but in the meantime, if you build canu from tip and replace the utgcns binary, it should address your crash.

[skoren]

I saw a message about building canu but din't see the post here, I assume you resolved the issue? It looked like an older version of git that couldn't initialize the submodules (marbl/canu#1747). It should be possible to run git submodule update --recursive to get the modules up to date to build canu.

[bernardo1963]

Dear Sergey,
Thank you for your help and care. I eventually succeeded, and got my verkko assembly. My server has CentOS7, which has an old gcc (4.8.5) and an old git (1.8.3.1); I think both cause errors while compiling canu. As others may stumble with the similar problems, I copied what I did below:

mkdir  /home6/tools/temp_git2
cd  /home6/tools/temp_git2
scl enable devtoolset-9 bash
gcc --version  
#gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2)
scl enable rh-git227 bash  
#git version 2.27.0
git clone https://github.com/marbl/canu.git
#remove the original unitigConsensus.C
rm /home6/tools/temp_git2/canu/src/utgcns/unitigConsensus.C
#now move the new commit of  unitigConsensus.C to the same folder
cd /home6/tools/temp_git2/canu/src
make -j 32
#find the executable utgcns inside canu
cd /home6/tools/temp_git2/canu
find . -type f  -name "utgcns" -exec ls -lh {} \;
#it is here:  /home6/tools/temp_git2/canu/build/bin/utgcns
#Here are the binaries I should replace inside verkko
mv /home6/tools/miniconda3/envs/verkko/lib/verkko/bin/utgcns /home6/tools/miniconda3/envs/verkko/lib/verkko/bin/utgcns.old
mv /home6/tools/miniconda3/pkgs/verkko-2.1-h45dadce_0/lib/verkko/bin/utgcns  /home6/tools/miniconda3/pkgs/verkko-2.1-h45dadce_0/lib/verkko/bin/utgcns.old
#copying the new utgcns (commit) to verkko
cp -p /home6/tools/temp_git2/canu/build/bin/utgcns   /home6/tools/miniconda3/envs/verkko/lib/verkko/bin/
cp -p /home6/tools/temp_git2/canu/build/bin/utgcns   /home6/tools/miniconda3/pkgs/verkko-2.1-h45dadce_0/lib/verkko/bin/
#now verkko run w/o errors

[skoren]

Thanks for the update.