Job groups and slurm jobarray

Create issue
Issue #676 new
jprnz created an issue

I am working on a workflow that will be submitting thousands of jobs to slurm and ought to be run via a jobarray. I was wondering if this might be a good use-case for the up coming job groups?

Currently I am having a difficult time coming up with a effective solution. What would be helpful would be to have, for each rule or grouped-rule, all job scripts available and able to be iterated over. I am not sure if this is similar / do-able with the planned groups feature, but something like this, in conjunction with recently added cluster flexibility, might make utilizing jobarrays more friendly.

Maybe I am overlooking some existing functionality to achieve this kind of behaviour?

A jobarray a method to serializes job submissions efficiently and requires only one call to sbatch. Upon execution sbatch passes an index (via env) to the executing command and typically this index is used to vary parameters for each job.

A typical scenario might look something like:

From the command line

sbatch --array 0-2 submission-script.sbatch

submission-script.sbatch

#!/bin/bash
#SBATCH --job-name myjobarray
index=$SLURM_ARRAY_JOB_ID
filelist=(a.txt b.txt c.txt)
mycommand ${filelist[$index]}

Thanks!

Comments (2)

  1. Ha Le

    I think snakemake could detect when it need to run the same rule for multiple sets of input and output. When this is the case, snakemake would write a new jobscript. This file will just look at the environment variable $SLURM_ARRAY_TASK_ID to pick one of the available jobscripts for execution. I'm willing to write the code for this but before I put more time into this, what is your opinion on this matter, @johanneskoester ? Do you have any better idea?

  2. Johnathan D

    I second this implementation of array jobs (PBS/SLURM etc.). I have recently had an increase in errors as:

    sbatch: error: slurm_receive_msg: Socket timed out on send/recv operation
    

    on our local cluster. This sometimes occurs when Snakemake is submitting 100s of jobs while also there are other users submitting a lot of jobs. Implementing array jobs would reduce the likelihood of this occurring significantly as only a single sbatch call would be needed.

    Interestingly, when these slurm send/recv errors occur even though the shell process has completed on the compute node the slurm_script and the python command on the compute node run until timeout. Snakemake then indefinitely keeps on checking for the status of this submitted job and never finishes. This is without drmaa. Using a --cluster-status script might catch this error but I think a possibility of array jobs would be an improvement. I have also used HPCs where array jobs are given precedence and have a larger concurrent job limit which did limit me when using Snakemake.

    So I agree with @hqle and wonder what do you think @johanneskoester ?

  3. Log in to comment