(core dumped)jgi_summarize_bam_contig_depths

[Corentin Hochart]

Hi,

I'm currently trying to generate a depth file from ~ 460 bam files (~340G - yeah it's a lot) with jgi_summarize_bam_contig_depths and i got this error:

(core dumped)jgi_summarize_bam_contig_depths --outputDepth depth.txt --pairedContigs paired.txt $mount_dir/bwa/*.bam

For info I'm using this script on SLURM managed cluster on 12T RAM node + bam files are stored on ceph stockage plateform and access with goofys soft.

I'm aware that I have a huge dataset and the problem must come from there but if you have an other idea I'll take it...

Regards, Corentin

[Rob Egan]

HI Corentin,

That's going to be hard to diagnose without any more information. That is a lot of bam files but I've run terabytes through that code without issue and I just tested running 1000 of the test bam files too. Are all the bam files sorted? How soon does it return the error (seconds or hours)?

Some options: 1) compile the debug version to get a stack trace and/or get it from the coredump, but the optimized version probably does not have enough symbols to diagnose the issue.

scons DEBUG=1

gdb jgi_summarize_bam_contig_depths core

2) one or more of your bam files may be corrupted, so try running one at a time and see if any of them fail individually.

3) If the bam files have a huge header, you may be running out of memory if you also have lots of cores on this machine, so reduce the parallelism by setting the environmental variable OMP_NUM_THREADS to a smaller number

[Corentin Hochart]

Hi Rob,

Thanks for the reply. Terabytes? Good to hear that. Well after checking all the bam files are correctly sorted. And the error return after around 30min.

I'm actually using 100 threads for a total of 11.966.583 sequences in the header... So I will try with a smaller number of threads and also check one by one my bam files to see if any of them is corrupted.

I will keep you updated.

[Corentin Hochart]

Hi Rob,

The MetaBAT version deployed on our HPC was in fact a precompiled one. After testing your different advises, our computer engineer prepared a version compiled by him and the problem was solved.

Thank you for your help.

Best Corentin