stop going upward on DAG - executing with limited space

Create issue
Issue #1147 new
Elena Grassi created an issue

Hi, I am wondering if I missed an option or not since it seems strange that I am the first one facing this issue. I am just checking out if the feature that I would need right now could be useful to others, if that's the case I could work at a PR.

Right now I have limited space (1Tb) on the computational node and I do no have any easy way to expand it. Being in the development phase of multiple projects I have several pipelines that are being built...usually we copy the starting files (eg. fastq) there and then start a pipeline (that for examples ends up in somatic mutation calls) - I am marking as many files as possible as temp() and deleting the fastq when the whole pipeline has ended, but if I add a rule (eg. to plot something new, to test a new hypothesis working on the somatic calls for multiple samples all together) I do not have any way to stop snakemake going up in the dag of dependencies and obviously complaining that the original fastq are not there anymore.

I am reasoning/using several caveats: 1- touching fastq and other temp files setting the right date 2- using --allowed-rules 3- splitting different portions of the pipelines without linking them via subworkflows, removing everything save the last files and then going on the next step "manually"

N.3 is clearly defeating the whole purpose :) N.1 is ok and can be made automatic via snakemake rules but requires a lot of "empty" code in my snakemakes - maybe this could be generalized, i.e. a general rule that removes and touches everything that has been produced with a given target...but still I am not really a fan of the idea of having dummy and real files mixed

N.2 is not suggested by the help itself and if half of the pipeline has legitimately to be re-run it could become cumbersome - does it make sense to add an option to tell snakemake to stop at a certain rule its path up in the DAG looking for missing or out of date dependencies?

Sorry for the long description, I hope at least that the issue I am facing is clear :)

Comments (0)

  1. Log in to comment