1. Juan Cabral
  2. poopy

Overview

HTTPS SSH

Poopy - map-reduce engine over AMQP

Install

From PYPI

  1. Install rabitMQ
  2. pip install poopy (please use virtualenv)
  3. Download this file https://bitbucket.org/leliel12/poopy/raw/tip/example/iris.arff?at=default
  4. Download this file https://bitbucket.org/leliel12/poopy/raw/tip/example/randomforest.py and put in the same directory of iris.arff

From REPO

  1. Install rabitMQ
  2. Clone this https://bitbucket.org/leliel12/poopy repo
  3. pip install -e .
  4. The archives iris.arff and randomforest.py are inside in examples/

Running

  1. Open two consoles (consoleA, consoleB)
  2. In consoleB run poopy deploy amqp://localhost
  3. In consoleA execute poopy upload amqp://localhost path/to/iris.arff poopFS://iris.arff now your file are uploaded to the "distributed file sistem"
  4. In consoleA run poopy run amqp://localhost path/to/randomforest.py Script out
  5. Your output model are serialized in out/localtime

Reading the model

In python console

from poopy import serializers

with open("out/file") as fp:
    model = serializers.load(fp)

TODO:

  • Implement correctly timeouts.
  • Implement an error exchange.
  • Real distributed file system.
  • More than one map or reduce.
  • More output formats than b64-pkl.
  • Not use the central node as main memory.
  • Reorder some modules into packages