snakemake uses an extremely high amount of memory when it throws an exception
I have a pipeline that I'm trying to call on ~2000 samples. The pipeline is pretty straightforward and can be reduced down to about 5 steps to recreate this issue. The problem is that when one of the steps throws an exception (file missing or something) snakemake takes an extremely long time to start printing the errors and will start eating up gigs and gigs of memory. A very small pipeline that's set to run on 2000 samples will sit and not give any output and work its way up to about 20Gb of memory usage before starting to output any reasonable output.
I love using snakemake but I'm wondering how well it's supposed to scale for a simple pipeline on say 2000 individuals?
We've developed a number of pipelines with a handful of individuals but, now as we try to scale we're having more and more trouble getting snakemake to behave. Any help would be appreciated. Thank you!