Source

bubble-economy / pbs.py

Full commit
#!/usr/bin/env python
# encoding: utf-8
"""
pbs.py

ultra thin subprocess-based qsub wrapper
"""

import subprocess

class Qsub(object):
    """
    from http://nf.nci.org.au/facilities/userguide/
    
    qsub options of note:

    -P project
    The project which you want to charge the jobs resource usage to. The default project is specified by the PROJECT environment variable.
    -q queue
    Select the queue to run the job in. The queues you can use are listed by running nqstat.
    -l walltime=??:??:??
    The wall clock time limit for the job. Time is expressed in seconds as an integer, or in the form:
    [[hours:]minutes:]seconds[.milliseconds] 
    System scheduling decisions depend heavily on the walltime request - it is always best to make as accurate a request as possible.
    -l vmem=???MB
    The total (virtual) memory limit for the job - can be specified with units of "MB" or "GB" but only integer values can be given. There is a small default value.
    Your job will only run if there is sufficient free memory so making a sensible memory request will allow your jobs to run sooner. A little trial and error may be required to find how much memory your jobs are using - nqstat lists jobs actual usage.
    -l ncpus=?
    The number of cpus required for the job to run on. The default is 1. 
    -lncpus=N - If the number of cpus requested, N, is small (currently 8 or less on NF systems) the job will run within a single shared memory node. If the number of cpus specified is greater, the job will (probably) be distributed over multiple nodes. Currently on NF systems, these larger requests are restricted to multiples of 8 cpus. 
    -lncpus=N:M - This form requests a total of N cpus with (a multiple of) M cpus per node. Typically, this is used to run shared memory jobs where M=N and N is currently limited to 8 on NF systems.
    -l jobfs=???GB
    The requested job scratch space. This will reserve disk space, making it unavailable for other jobs, so please do not over estimate your needs. Any files created in the $PBS_JOBFS directory are automatically removed at the end of the job. Ensure that you use integers, and units of mb, MB, gb, or GB.
    -l software=???
    Specifies licensed software the job requires to run. See the software for the string to use for specific software. The string should be a colon separated list (no spaces) if more than one software product is used.
    If your job uses licensed software and you do not specify this option (or mis-spell the software), you will probably receive an automatically generated email from the license shadowing daemon (see man lsd), and the job may be terminated. You can check the lsd status and find out more by looking at the URL mentioned in man lsd.

    -l other=???
    Specifies other requirements or attributes of the job. The string should be a colon separated list (no spaces) if more than one attribute is required. Generally supported attributes are:
    iobound - the job should not share a node with other IO bound jobs
    mdss - the job requires access to the MDSS (usually via the mdss command). If MDSS is down, the job will not be started.
    pernodejobfs - the job's jobfs resource request should be treated as a per node request. Normally the jobfs request is for total jobfs summed over all nodes allocated to the job (like vmem). Only relevant to distributed parallel jobs using jobfs.
    You may be asked to specify other options at times to support particular needs or circumstances.
    -r y
    Specifies your job is restartable, and if the job is executing on a node when it crashes, the job will be requeued. Both resources used by and resource limits set for the original job will carry over to the requeued job. Hence a restartable job must be checkpointing such that it will still be able to complete in the remaining walltime should it suffer a node crash.
    The default is that jobs are assumed to not be restartable. Note that regardless of the restartable status of a job, time used by jobs on crashed nodes is charged against the project they are running under, since the onus is on users to ensure minimum waste of resources via a checkpointing mechanism which they must build into any particularly long running codes.

    -wd
    Start the job in the directory from which it was submitted. Normally jobs are started in the user's home directory
    """
    def __init__(self, l_args=None, **q_args):
        #things I rarely change:
        self.q_args = {'q': 'normal'}
        self.l_args = {
            'ncpus': 1 #unless you're doing python OpenMPI. Weirdo.
        }
        self.q_args.update(q_args)
        self.l_args.update(l_args or {})
        
    def __call__(self, script_command, extra_l_args=None, **extra_q_args):
        q_args = self.q_args.copy()
        l_args = self.l_args.copy()
        q_args.update(extra_q_args or {})
        l_args.update(extra_l_args or {})                                                                                                                                                                                                                                                                                                                                        
        #options of the -l form can be concatenated
        args = ['qsub']
        
        if l_args:
            l_arg_string = '-l' + ','.join(
                ['='.join([k, str(v)]) for k, v in l_args.iteritems()]
            )
            args.append(l_arg_string)
        for k,v in q_args.iteritems():
            args.extend(["-"+k, str(v)])
        #Now, our command is a list, so we escape-quote it.
        actual_command = subprocess.list2cmdline(script_command)
        args.append("bin/run_command_with_args.sh")
        args.append("-v PBS_COMMAND='%s'" % actual_command)
        res = subprocess.check_output(args).strip()
        return res