plow /

Filename Size Date modified Message
examples
tests
225 B
Added tag 0.3 for changeset fa24634a9c72
723 B
Typo
34.3 KB
Add license
1.9 KB
Explain the remote env setup
6.6 KB
Better explain in comment
18.5 KB
Better error message for invalid targets

Plow workflow execution engine

Plow is a workflow execution engine inspired by Makeflow.

Features:
  • deps aware execution
  • multiprocess execution
  • incremental transition from imperative workflow execution with memo.py
  • remote execution (can setup the remote env with execnet xspec syntax)

Installation

Plow is a Python (2.6+) script and depends on:

To install download the latest plow.py and memo.py and make them executable:

wget https://bitbucket.org/naufraghi/plow/raw/tip/{memo,plow}.py
chmod +x {memo,plow}.py

Example usage

Step 1

Write your workflow some language, for example in bash:

$ cat step1.sh
#!/bin/bash
res = ()
for x in 1 2 3 4; do
  echo $x > $x.res
  res+=($x.res)
done
cat ${res[@]} | sort | tail -n 1 > res.max

Step 2

Use memo.py to memoize results (execute only if some dep changed):

$ cat step2.sh
#!/bin/bash
res = ()
for x in 1 2 3 4; do
  ./memo.py "echo $x > $x.res"
  res+=($x.res)
done
./memo.py "cat ${res[@]} | sort | tail -n 1 > res.max"

Step 3

Echo out all the commands if the memo.py overhead is big:

$ cat step3.sh
#!/bin/bash
res = ()
for x in 1 2 3 4; do
  echo "echo $x > $x.res"
  res+=($x.res)
done
echo "cat ${res[@]} | sort | tail -n 1 > res.max"

And convert the commands in a plow file:

$ bash step3.sh > step3.commands
$ ./memo.py -p -l step3.commands -o step3.plow
$ ./plow.py step3.plow res.max
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.