HTTPS SSH

QDO

QDO (kew-doo) is a lightweight high-throughput queuing system for workflows that have many small tasks to perform. It is designed for situations where the number of tasks to perform is much larger than the practical limits of the underlying batch job system. Its interface emphasizes simplicity while maintaining flexibility.

Examples

Running many tasks using python interface

#- Create a queue object
import qdo
q = qdo.create('EchoChamber')

#- Add 100 tasks
for i in range(100):
    q.add('echo Hello ' + str(i))

#- Or use add_multiple for faster loading of many many task
tasks = ['echo Wow ' + str(i) for i in range(10000)]
q.add_multiple(tasks)

#- submit 8 batch jobs to process these tasks until none are left
q.launch(1, jobs=8)

#- or pack 8 workers (non-mpi) into a single batch job
q.launch(8)

#- Even after the jobs are launched, you can add more tasks to the queue
for i in range(10):
    q.add('echo Goodbye ' + str(i))

#- Check the progress
q.status()

Loading commands from a file using command line interface

#- Generate a list of commands to perform
ls *.tif | awk '{print "awesome_analysis " $1}' > commands.txt

#- Load those commands into a queue called ImgQueue
qdo load ImgQueue commands.txt

#- List all known queues and their status
qdo list

#- launch 10 batch jobs to process the commands in ImgQueue
qdo launch ImgQueue 1 --jobs 10 --taskcores=1

#- launch 10 workers in a job  jobs to process the commands in ImgQueue
qdo launch ImgQueue 10 --taskcores=1

Queuing parameters instead of executables

Code to load the queue:

import qdo
import os, glob

q = qdo.create('ImgQueue')
imgdir = os.getcwd()
for filename in glob.glob('*.tif'):
    params = dict(imgdir=imgdir, infile=filename, outfile=filename+'.out')
    q.add(params)

Client code to process parameters from the queue:

import qdo

def process_image(imgdir, infile, outfile):
    print "Convert %s to %s in %s" % (infile, outfile, imgdir)

q = qdo.connect('ImgQueue')
while True:
    task = q.get(timeout=10)
    #- task.task will be the params dictionary
    if task is not None:
        try:
            print "processing", task.task
            process_image(**task.task)
            print "yay!"
            task.set_state(qdo.Task.SUCCEEDED)
        except:
            print "oops"
            task.set_state(qdo.Task.FAILED, err=1)
    else:
        #- no more tasks in queue so break
        break

More than one copy of the client code can run simultaneously. The queue guarantees that the same task will only be sent to one client.

Other Features

Check the status of tasks in the queue

python:  q.status()
command: qdo status QueueName

Get details about every task in the queue

python:  q.tasks()
command: qdo tasks QueueName [--verbose]

Get details for only tasks in a given state

python:  q.tasks(state=qdo.Task.FAILED)
command: qdo tassk QueueName --state Failed

Pause/resume - lets running tasks cleanly finish but won't hand out new ones

python:  q.pause()
         q.resume()
command: qdo pause QueueName
         qdo resume QueueName

Retry failed tasks:

python:  q.retry()
command: qdo retry QueueName

Rerun all tasks (succeeded or failed):

python:  q.rerun()
command: qdo rerun QueueName --force

Add a task with a specified priority (default 0, higher priority runs first):

python:  q.add(task, priority=100)
command: qdo add QueueName "echo very important" --priority 100

Delete a queue:

python:  q.delete()
command: qdo delete QueueName --force

Task dependencies and priorities are also supported; see doc/dependencies.md and doc/priorities.md .

Configuration

Configuration parameters of qdo can be set through environment variables and a qdorc file. If a value is set both in the qdorc file and an environment variable, the environment prevails.

When invoked, qdo will try to find the qdorc file at (in this order): - if set, at $QDORC_LOCATION - if $QDO_DIR set, at $QDO_DIR/qdorc - at ~/.qdo/qdorc

If the file is not found, a sample qdroc will be created at the configured location. Modify this file for custom configuration.

Upcoming Features

Save/Restore: dump a queue to an archive file, or restore a queue from a previously dumped archive.

Easier log file management.

Running Test Suite

cd test
python -m unittest discover

Authors

qdo is developed by Stephen Bailey at Lawrence Berkeley National Lab, with additional contributions from Monte Goode, Dan Gunter and Gonzalo Rodrigo. It is released under the BSD 3-clause open source licence, see LICENSE.