plow /

Filename Size Date modified Message
225 B
Added tag 0.3 for changeset fa24634a9c72
723 B
34.3 KB
Add license
1.9 KB
Explain the remote env setup
6.6 KB
Better explain in comment
18.5 KB
Better error message for invalid targets

Plow workflow execution engine

Plow is a workflow execution engine inspired by Makeflow.

  • deps aware execution
  • multiprocess execution
  • incremental transition from imperative workflow execution with
  • remote execution (can setup the remote env with execnet xspec syntax)


Plow is a Python (2.6+) script and depends on:

To install download the latest and and make them executable:

chmod +x {memo,plow}.py

Example usage

Step 1

Write your workflow some language, for example in bash:

$ cat
res = ()
for x in 1 2 3 4; do
  echo $x > $x.res
cat ${res[@]} | sort | tail -n 1 > res.max

Step 2

Use to memoize results (execute only if some dep changed):

$ cat
res = ()
for x in 1 2 3 4; do
  ./ "echo $x > $x.res"
./ "cat ${res[@]} | sort | tail -n 1 > res.max"

Step 3

Echo out all the commands if the overhead is big:

$ cat
res = ()
for x in 1 2 3 4; do
  echo "echo $x > $x.res"
echo "cat ${res[@]} | sort | tail -n 1 > res.max"

And convert the commands in a plow file:

$ bash > step3.commands
$ ./ -p -l step3.commands -o step3.plow
$ ./ step3.plow res.max
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.