Source

plow /

Filename Size Date modified Message
examples
tests
135 B
34.3 KB
1.9 KB
6.5 KB
17.1 KB

Plow workflow execution engine

Plow is a workflow execution engine inspired by Makeflow.

Features:
  • deps aware execution
  • multiprocess execution
  • incremental transition from imperative workflow execution with memo.py
  • remote execution (with execnet xspec syntax)

Installation

Plow is a Python (2.6+) script and depends on:

To install download the latest plow.py and memo.py and make them executable:

wget https://bitbucket.org/naufraghi/plow/raw/tip/{memo,plow}.py
chmod +x {memo,plow}.py

Example usage

Step 1

Write your workflow some language, for example in bash:

$ cat step1.sh
#!/bin/bash
res = ()
for x in 1 2 3 4; do
  echo $x > $x.res
  res+=($x.res)
done
cat ${res[@]} | sort | tail -n 1 > res.max

Step 2

Use memo.py to memoize results (execute only if some dep changed):

$ cat step2.sh
#!/bin/bash
res = ()
for x in 1 2 3 4; do
  ./memo.py "echo $x > $x.res"
  res+=($x.res)
done
./memo.py "cat ${res[@]} | sort | tail -n 1 > res.max"

Step 3

Echo out all the commands if the memo.py overhead is big:

$ cat step3.sh
#!/bin/bash
res = ()
for x in 1 2 3 4; do
  echo "echo $x > $x.res"
  res+=($x.res)
done
echo "cat ${res[@]} | sort | tail -n 1 > res.max"

And convert the commands in a plow file:

$ bash step3.sh > step3.commands
$ ./memo.py -p -l step3.commands -o step3.plow
$ ./plow.py step3.plow res.max