Rule-specific cluster resource requirements

Create issue
Issue #56 resolved
Martin Aryee created an issue

We often have a combination of small resource rules (a few mins, <10MB RAM) and large resource rules (hours/days and >10GB RAM). It would be helpful to be able to specify the cluster queue and resource requirements in the rule definition. This proposal is similar to Issue #44 (Allow "local" jobs when submitting to a cluster).

My solution at the moment is to use the --cluster option to specify the largest job's queue and resource requirements, but this results in our 'big job' queue being filled with a large number of tiny jobs that don't need to be there. One downside with this proposal is that it would make the Snakefile less portable across cluster systems.

Comments (3)

  1. Johannes Köster

    Welcome! Actually there is already a facility to achieve that. You can encode additional information into the params definition of a rule. This can be referenced from the specified cluster command. This feature was unfortunately undocumented so far. I just added some explaining statements to the end of the section "Executing Snakefiles" in the documentation.

    Does this help? Thanks for pointing this out! Johannes

  2. Martin Aryee reporter

    That functionality perfectly addresses the issue and makes Snakemake even more useful! Thanks very much!

  3. Log in to comment