line 118: 2931 Segmentation fault (core dumped)

[Former user]

while I was binning I met this problem. I wrote "./runMetaBat.sh /home/ls/wbs/final.contigs.fa /home/ls/metabat/final.contigs.bowtie.sort.bam" metabat feedback "Skipping jgi_summarize_bam_contig_depths as final.contigs.fa.depth.txt already exists Executing: 'metabat2 --inFile /home/ls/wbs/final.contigs.fa --outFile final.contigs.fa.metabat-bins/bin --abdFile final.contigs.fa.depth.txt' at Fri Apr 10 11:03:35 CST 2020 MetaBAT 2 (v2.12.1) using minContig 2500, minCV 1.0, minCVSum 1.0, maxP 95%, minS 60, and maxEdges 200. ./runMetaBat.sh: line 118: 2931 Segmentation fault (core dumped) $MB $metabatopts --inFile $assembly --outFile $outname --abdFile ${depth} " I have tried several times, met line 118 problem everytime I wonderend it might be the memery problem, but not that sure, so I am here for help! thanks a lot!

[Rob Egan]

This is hard to diagnose at a distance without looking at the core file that was dumped.

It could be memory, though I would expect an error message if it ran out and the allocation failed. It could also be that a previous run failed to write the entire depth.txt file (that your log says it is skipping since it already exists), so if that file is truncated or otherwise malformed the code could crash while reading it.

Please try again after removing final.contigs.fa.depth.txt

And run with the -v option and keep the entire log file. If it crashes again, please post that log file.

Also, if there is a core file, then a stack trace might help.

‌

gdb -q -n -ex bt -batch /path/to/metabat2 /path/to/corefile

‌

[Remi Maglione]

Hello Rob,
I experienced the same issue today. It’s an instant crash. I’m working on a server with 64 cores and 500Go memory. Here my entire log:

metabat2 -v --inFile ../7.pilon/polished_assembly.fasta --abdFile polished_assembly.fasta.depth.txt -o bin
MetaBAT 2 (v2.12.1) using minContig 2500, minCV 1.0, minCVSum 1.0, maxP 95%, minS 60, and maxEdges 200.
[00:00:00] Finished reading 34 contigs and 2 coverages from polished_assembly.fasta.depth.txt
[00:00:00] Number of target contigs: 14 of large (>= 2500) and 19 of small ones (>=1000 & <2500).
[00:00:00] Finished TNF calculation.
[00:00:00] Finished Preparing TNF Graph Building [pTNF = 69.30]
[00:00:00] Finished Building TNF Graph (2 vertices and 1 edges) [0.1Gb / 504.9Gb]
Segmentation fault

Any help would be welcome.

Best,

Rémi

[Rob Egan]

So the next stage is reading in the abundance file. If that is malformed, then a segfault is possible. Please show the log on how was that file generated. Was there an error in the previous stage that processed the bam files? I suspect it is corrupted or not the expected format

Please post the first two lines and last line of this file: polished_assembly.fasta.depth.txt

Also, I recommend running with a newer version. 2.12.1 is over 3 years old.

And in your next attempt please run with the --verbose option set.

It might also help to get your system to drop a core file and include the stack trace from it.

[Remi Maglione]

Hello Rob,
I processed the bam files with runMetaBat.sh. With --verbose, here what I get:

Executing: 'jgi_summarize_bam_contig_depths --outputDepth polished_assembly.fasta.depth.txt --pairedContigs polished_assembly.fasta.paired.txt --minContigLength 1000 --minContigDepth 1  ../8.minimap2_mapping/r1t1_2017AT_NP_long_sorted.bam ../9.bwa_mapping/illumina_sorted.bam' at Sun Apr 26 10:24:24 EDT 2020
Output depth matrix to polished_assembly.fasta.depth.txt
Output pairedContigs lower triangle to polished_assembly.fasta.paired.txt
minContigLength: 1000
minContigDepth: 1
Output matrix to polished_assembly.fasta.depth.txt
Opening 2 bams
Consolidating headers
Allocating pairedContigs matrix: 0 MB over 1 threads
Processing bam files
Thread 0 finished: r1t1_2017AT_NP_long_sorted.bam with 3771645 reads and 4699 readsWellMapped
Thread 0 finished: illumina_sorted.bam with 20886426 reads and 193589 readsWellMapped
Creating depth matrix file: polished_assembly.fasta.depth.txt
Closing most bam files
Creating pairedContigs matrix file: polished_assembly.fasta.paired.txt
Closing last bam file
Finished
Finished jgi_summarize_bam_contig_depths at Sun Apr 26 10:26:37 EDT 2020
Creating depth file for metabat at Sun Apr 26 10:26:37 EDT 2020
Executing: 'metabat2  --verbose -t 60 --inFile ../7.pilon/polished_assembly.fasta --outFile polished_assembly.fasta.metabat-bins60/bin --abdFile polished_assembly.fasta.depth.txt' at Sun Apr 26 10:26:37 EDT 2020
MetaBAT 2 (v2.12.1) using minContig 2500, minCV 1.0, minCVSum 1.0, maxP 95%, minS 60, and maxEdges 200.
[00:00:00] Finished reading 34 contigs and 2 coverages from polished_assembly.fasta.depth.txt
[00:00:00] Number of target contigs: 14 of large (>= 2500) and 19 of small ones (>=1000 & <2500).
[00:00:00] Finished TNF calculation.
[00:00:00] Finished Preparing TNF Graph Building [pTNF = 69.70]
[00:00:00] Finished Building TNF Graph (2 vertices and 1 edges) [0.1Gb / 504.9Gb]
/bio/maglione/miniconda3/bin/runMetaBat.sh: line 118: 30478 Segmentation fault      $MB $metabatopts --inFile $assembly --outFile $outname --abdFile ${depth}

The previous log was an attempt to run the metabat2 step (already with -v option) with the depth file yielded with runMetaBat.sh

---

Here the two 1rst lines of polished_assembly.fasta.depth.txt

contigName      contigLen       totalAvgDepth   r1t1_2017AT_NP_long_sorted.bam  r1t1_2017AT_NP_long_sorted.bam-var      illumina_sorted.bam     illumina_sorted.bam-var
contig_10_segment0_pilon_pilon  4646    290.204 0.0108986       0.0107822       290.193 219917

and the two last lines:

contig_8_segment0_pilon_pilon   8140    50.0173 0.15194 0.174425        49.8653 1634.37
contig_9_segment0_pilon_pilon   9943    50.9246 0.113755        0.114099        50.8109 18142.2

‌

---

version 2.12.1 was the only version I was able to install (through conda) because the current installation from source always crash at the ‘make’ step :

[ 76%] Linking CXX executable metabat1
/usr/bin/ld: BFD version 2.20.51.0.2-5.48.el6 20100205 internal error, aborting at reloc.c line 443 in bfd_get_reloc_size

/usr/bin/ld: Please report this bug.

collect2: error: ld returned 1 exit status
make[2]: *** [src/metabat1] Error 1
make[2]: *** Deleting file `src/metabat1'
make[1]: *** [src/CMakeFiles/metabat1.dir/all] Error 2
make: *** [all] Error 2

‌

---

It might also help to get your system to drop a core file and include the stack trace from it.

Sorry Rob, but my skills end up here I know you previously suggested that command line gdb -q -n -ex bt -batch /path/to/metabat2 /path/to/corefile, I didn’t find the core dump file. Where can I find it ?

[Rob Egan]

So first the compile issue. The top Google hit to that problem indicates that your installation of binutils is the issue and that it is over 10 years old. metabat requires a modern compiler environment, so it does not surprise me that the old linker on yours system does not work with it. You’ll need to update your install of binutils (or use our docker image). Conda is not supported anymore.

You might be able to have your system drop a core file if you set:

ulimit -c unlimited

… but this is likely to be of limited use if the version you have does not include the debug symbols.

The input files look okay, but unfortunately I cannot help with this remotely if you cannot build a newer version.

If you are okay sharing your files with me I can try to see what may be happening polished_assembly.fasta & polished_assembly.fasta.depth.txt

Feel free to DM me if you want me to take a look at your data: [e-mail]

‌

[Remi Maglione]

Hello Rob,

Is it very tough for me to compile metabata without admin rights (I spent my all day trying to compile, I had even had to link all the boost and gcc from my conda env but everything failed and I’m about to give up on this)
here what I tried after the update of binutils

cd metageno/tool/berkeleylab-metabat-a101cde67177/build/
(base) -bash-4.1$ /bio/maglione/miniconda2^H3/bin/cmake -DCMAKE_INSTALL_PREFIX=/bio/maglione/tool ..
-- The C compiler identification is GNU 4.4.7
-- The CXX compiler identification is GNU 5.1.0
-- Check for working C compiler: /usr/lib64/ccache/cc
-- Check for working C compiler: /usr/lib64/ccache/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/lib64/ccache/c++
-- Check for working CXX compiler: /usr/lib64/ccache/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Installing Release MetaBAT into /bio/maglione/tool
CMake Error at /bio/maglione/miniconda3/share/cmake-3.14/Modules/FindBoost.cmake:2132 (message):
  Unable to find the requested Boost libraries.

  Boost version: 1.41.0

  Boost include path: /usr/include

  Detected version of Boost is too old.  Requested version was 1.55 (or
  newer).
Call Stack (most recent call first):
  src/CMakeLists.txt:4 (find_package)


-- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
-- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)
CMake Warning at src/CMakeLists.txt:16 (message):
  OpenMP must be available to your compiler or it will be REALLLLLLLLLLY
  slow: /usr/lib64/ccache/cc /usr/lib64/ccache/c++


Boost cmake dir:
Boost include: /usr/include
Boost libraries:
Boost lib ver:1_41
Boost lib dir:/bio/maglione/miniconda3/lib
-- Configuring incomplete, errors occurred!
See also "/bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/CMakeFiles/CMakeOutput.log".
See also "/bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/CMakeFiles/CMakeError.log".

But here is what I have in my conda env:

binutils                  2.34
binutils_linux-64         2.34
boost                     1.68.0          py37h8619c78_1001    conda-forge
boost-cpp                 1.68.0            h11c811c_1000    conda-forge
cmake                     3.14.0               h52cb24c_0    anaconda

Since boost is not the version I have and cmake keep looking at /usr/lib64/ccache/cc for c and c ++, I tried to force different path on cmake with:

‌

/bio/maglione/miniconda2^H3/bin/cmake -DCMAKE_INSTALL_PREFIX=/bio/maglione/tool -D CMAKE_C_COMPILER=/bio/maglione/gcc-5.1.0/bin/gcc -D CMAKE_CXX_COMPILER=/bio/maglione/gcc-5.1.0/bin/c++ -D Boost_INCLUDE_DIR=/bio/maglione/miniconda2^H3/include ..
-- The C compiler identification is GNU 5.1.0
-- The CXX compiler identification is GNU 5.1.0
-- Check for working C compiler: /bio/maglione/gcc-5.1.0/bin/gcc
-- Check for working C compiler: /bio/maglione/gcc-5.1.0/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /bio/maglione/gcc-5.1.0/bin/c++
-- Check for working CXX compiler: /bio/maglione/gcc-5.1.0/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Installing Release MetaBAT into /bio/maglione/tool
-- Boost version: 1.68.0
-- Found the following Boost libraries:
--   program_options
--   filesystem
--   system
--   graph
--   serialization
--   iostreams
--   regex
-- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
-- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)
CMake Warning at src/CMakeLists.txt:16 (message):
  OpenMP must be available to your compiler or it will be REALLLLLLLLLLY
  slow: /bio/maglione/gcc-5.1.0/bin/gcc /bio/maglione/gcc-5.1.0/bin/c++


Boost cmake dir:
Boost include: /bio/maglione/miniconda3/include
Boost libraries:/bio/maglione/miniconda3/lib/libboost_program_options.a/bio/maglione/miniconda3/lib/libboost_filesystem.a/bio/maglione/miniconda3/lib/libboost_system.a/bio/maglione/miniconda3/lib/libboost_graph.a/bio/maglione/miniconda3/lib/libboost_serialization.a/bio/maglione/miniconda3/lib/libboost_iostreams.a/bio/maglione/miniconda3/lib/libboost_regex.a
Boost lib ver:1_68
Boost lib dir:/bio/maglione/miniconda3/lib
-- Configuring done
-- Generating done
-- Build files have been written to: /bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build

But I get this

make
Scanning dependencies of target zlib
[  4%] Creating directories for 'zlib'
[  8%] Performing download step (git clone) for 'zlib'
-- zlib download command succeeded.  See also /bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/contrib/zlib-prefix/src/zlib-stamp/zlib-download-*.log
[ 12%] No patch step for 'zlib'
[ 16%] No update step for 'zlib'
[ 20%] Performing configure step for 'zlib'
Checking for gcc...
Building static library libz.a version 1.2.11 with gcc.
Checking for size_t... Yes.
Checking for off64_t... Yes.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.
[ 24%] Performing build step for 'zlib'
[ 28%] Performing install step for 'zlib'
-- zlib install command succeeded.  See also /bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/contrib/zlib-prefix/src/zlib-stamp/zlib-install-*.log
[ 32%] Completed 'zlib'
[ 32%] Built target zlib
Scanning dependencies of target htslib
[ 36%] Creating directories for 'htslib'
[ 40%] Performing download step (git clone) for 'htslib'
-- htslib download command succeeded.  See also /bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/contrib/htslib-prefix/src/htslib-stamp/htslib-download-*.log
[ 44%] No patch step for 'htslib'
[ 48%] No update step for 'htslib'
[ 52%] Performing configure step for 'htslib'
configure.ac:26: error: possibly undefined macro: m4_esyscmd_s
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
make[2]: *** [contrib/htslib-prefix/src/htslib-stamp/htslib-configure] Error 1
make[1]: *** [CMakeFiles/htslib.dir/all] Error 2
make: *** [all] Error 2

I don’t understand this error and I tried to skip it by just relaunching a make:

make
[ 32%] Built target zlib
[ 36%] Performing configure step for 'htslib'
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for ranlib... /bio/maglione/miniconda3/bin/x86_64-conda_cos6-linux-gnu-ranlib
checking for grep that handles long lines and -e... /bin/grep
checking for C compiler warning flags... -Wall
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for _LARGEFILE_SOURCE value needed for large files... no
checking shared library type for unknown-Linux... plain .so
checking how to run the C preprocessor... gcc -E
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking for getpagesize... yes
checking for working mmap... yes
checking for gmtime_r... yes
checking for fsync... yes
checking for drand48... yes
checking whether fdatasync(int) is declared... no
checking for library containing log... -lm
checking for zlib.h... yes
checking for inflate in -lz... yes
checking for library containing recv... none required
checking for libdeflate.h... no
checking for libdeflate_deflate_compress in -ldeflate... no
configure: WARNING: GCS support not enabled: requires libcurl support
configure: WARNING: S3 support not enabled: requires libcurl support
checking whether PTHREAD_MUTEX_RECURSIVE is declared... yes
configure: creating ./config.status
config.status: creating config.mk
config.status: creating htslib.pc.tmp
config.status: creating config.h
[ 40%] Performing build step for 'htslib'
[ 44%] Performing install step for 'htslib'
[ 48%] Completed 'htslib'
[ 64%] Built target htslib
Scanning dependencies of target check_git_repository
[ 68%] Checking the git repository for changes...
Skipping creation of /bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/metabat_version.h as git is not available
[ 68%] Built target check_git_repository
Scanning dependencies of target metabat1
[ 72%] Building CXX object src/CMakeFiles/metabat1.dir/metabat1.cpp.o
[ 76%] Linking CXX executable metabat1
[ 76%] Built target metabat1
Scanning dependencies of target metabat2
[ 80%] Building CXX object src/CMakeFiles/metabat2.dir/metabat2.cpp.o
[ 84%] Linking CXX executable metabat2
[ 84%] Built target metabat2
Scanning dependencies of target jgi_summarize_bam_contig_depths
[ 88%] Building CXX object src/CMakeFiles/jgi_summarize_bam_contig_depths.dir/jgi_summarize_bam_contig_depths.cpp.o
[ 92%] Linking CXX executable jgi_summarize_bam_contig_depths
/bio/maglione/miniconda3/bin/ld: ../contrib/htslib-install/lib/libhts.a(bgzf.o): in function `mt_destroy':
/bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/contrib/htslib-prefix/src/htslib/bgzf.c:1431: undefined reference to `pthread_join'
/bio/maglione/miniconda3/bin/ld: ../contrib/htslib-install/lib/libhts.a(bgzf.o): in function `bgzf_thread_pool':
/bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/contrib/htslib-prefix/src/htslib/bgzf.c:1394: undefined reference to `pthread_create'
/bio/maglione/miniconda3/bin/ld: ../contrib/htslib-install/lib/libhts.a(thread_pool.o): in function `hts_tpool_init':
/bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/contrib/htslib-prefix/src/htslib/thread_pool.c:678: undefined reference to `pthread_mutexattr_init'
/bio/maglione/miniconda3/bin/ld: /bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/contrib/htslib-prefix/src/htslib/thread_pool.c:679: undefined reference to `pthread_mutexattr_settype'
/bio/maglione/miniconda3/bin/ld: /bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/contrib/htslib-prefix/src/htslib/thread_pool.c:681: undefined reference to `pthread_mutexattr_destroy'
/bio/maglione/miniconda3/bin/ld: /bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/contrib/htslib-prefix/src/htslib/thread_pool.c:695: undefined reference to `pthread_create'
/bio/maglione/miniconda3/bin/ld: ../contrib/htslib-install/lib/libhts.a(thread_pool.o): in function `hts_tpool_destroy':
/bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/contrib/htslib-prefix/src/htslib/thread_pool.c:945: undefined reference to `pthread_join'
/bio/maglione/miniconda3/bin/ld: ../contrib/htslib-install/lib/libhts.a(thread_pool.o): in function `hts_tpool_kill':
/bio/maglione/metageno/tool/berkeleylab-metabat-a101cde67177/build/contrib/htslib-prefix/src/htslib/thread_pool.c:971: undefined reference to `pthread_kill'
collect2: error: ld returned 1 exit status
make[2]: *** [src/jgi_summarize_bam_contig_depths] Error 1
make[1]: *** [src/CMakeFiles/jgi_summarize_bam_contig_depths.dir/all] Error 2
make: *** [all] Error 2

For an unknown reason htslib step pass this time but I get this new error, and it’s where I'm stuck right now.
I’ll try to dock metabat but without admin right, it’ll be hard for me to install docker. I’ll keep you posted.
Rémi

[Remi Maglione]

Hello Rob,
I found that docker could be install without root privilege https://docs.docker.com/engine/security/rootless/,
but it looks like it needs some system requirement (with with root privilege…):

curl -fsSL https://get.docker.com/rootless | sh
# Installing stable version 19.03.8
# Missing system requirements. Please run following commands to
# install the requirements and run this installer again.
# Alternatively iptables checks can be disabled with SKIP_IPTABLES=1

cat <<EOF | sudo sh -x
curl -o /etc/yum.repos.d/vbatts-shadow-utils-newxidmap-epel-7.repo https://copr.fedorainfracloud.org/coprs/vbatts/shadow-utils-newxidmap/repo/epel-7/vbatts-shadow-utils-newxidmap-epel-7.repo
yum install -y shadow-utils46-newxidmap
EOF

If you have any idea, it will be welcome, because I’m out of solutions here.
Best,
Rémi

[Rob Egan]

Hi Remi,

Sorry that you are going down the rabbit hole with this install. I can tell you that upgrading an old system like you have to a modern compile environment is a very tricky and problematic thing to do. The prerequisites we have for metabat are not unreasonable, but do require a OS install from around the past 5 years.

Unfortunately on your system, it looks like the autoconf tools and/or cmake are also very old and causing problems detecting the OpenMP support which is present in the g++ 5.1 compiler and also had problems executing the autoconf tools for the htslib package.

Your second attempt looks like it got close and if autotools and cmake were both upgraded in your conda environment, it might work:

/bio/maglione/miniconda2^H3/bin/cmake -DCMAKE_INSTALL_PREFIX=/bio/maglione/tool -D CMAKE_C_COMPILER=/bio/maglione/gcc-5.1.0/bin/gcc -D CMAKE_CXX_COMPILER=/bio/maglione/gcc-5.1.0/bin/c++ -D Boost_INCLUDE_DIR=/bio/maglione/miniconda2^H3/include ..

‌

If you don’t have access to a docker server, then there is no way I know of making that work.

If you are willing to pay for Amazon, then that could be a viable option. I think they still have an option for free time / trial.

It also looks like your fasta and depths table are not very large, so you might want to try running via docker on your desktop/laptop if you have admin access there.

I’ll also ping our conda expert to see if he is willing to build and push a new version to conda, but that may take a while.

‌

At this point, though, my best guess why the segfault that started this ticket is happening is because your system is very old and there may be a call (likely within the conda-build’s dependencies on OpenMP) that is triggering a bug in the old system libraries on your machine.

Again, if you want me to have a go at your data on my systems, just let me know in a separate email.