Source

plow /

Filename Size Date modified Message
examples
tests
225 B
Added tag 0.3 for changeset fa24634a9c72
723 B
Typo
34.3 KB
Add license
1.9 KB
Explain the remote env setup
6.6 KB
Better explain in comment
18.5 KB
Better error message for invalid targets

Plow workflow execution engine

Plow is a workflow execution engine inspired by Makeflow.

Features:
  • deps aware execution
  • multiprocess execution
  • incremental transition from imperative workflow execution with memo.py
  • remote execution (can setup the remote env with execnet xspec syntax)

Installation

Plow is a Python (2.6+) script and depends on:

To install download the latest plow.py and memo.py and make them executable:

wget https://bitbucket.org/naufraghi/plow/raw/tip/{memo,plow}.py
chmod +x {memo,plow}.py

Example usage

Step 1

Write your workflow some language, for example in bash:

$ cat step1.sh
#!/bin/bash
res = ()
for x in 1 2 3 4; do
  echo $x > $x.res
  res+=($x.res)
done
cat ${res[@]} | sort | tail -n 1 > res.max

Step 2

Use memo.py to memoize results (execute only if some dep changed):

$ cat step2.sh
#!/bin/bash
res = ()
for x in 1 2 3 4; do
  ./memo.py "echo $x > $x.res"
  res+=($x.res)
done
./memo.py "cat ${res[@]} | sort | tail -n 1 > res.max"

Step 3

Echo out all the commands if the memo.py overhead is big:

$ cat step3.sh
#!/bin/bash
res = ()
for x in 1 2 3 4; do
  echo "echo $x > $x.res"
  res+=($x.res)
done
echo "cat ${res[@]} | sort | tail -n 1 > res.max"

And convert the commands in a plow file:

$ bash step3.sh > step3.commands
$ ./memo.py -p -l step3.commands -o step3.plow
$ ./plow.py step3.plow res.max