Commits

Mathieu Bridon committed 2a8a908 Merge

Merge

Comments (0)

Files changed (23)

documentation/source/_static/adapter1.png

Added
New image

documentation/source/_static/dragndrop-anim.png

Added
New image

documentation/source/_static/dragndrop.png

Added
New image

documentation/source/_static/dragndrop2.png

Added
New image

documentation/source/_static/filter1.png

Added
New image

documentation/source/_static/newtube.png

Added
New image

documentation/source/_static/producer.png

Added
New image

documentation/source/_static/props.png

Added
New image

documentation/source/_static/tube_menu.png

Added
New image

documentation/source/_static/tutorial_tube.png

Added
New image

documentation/source/_static/two_branches.png

Added
New image

documentation/source/contents/modules/componentized/index.rst

+*****************
 PyF.Componentized
-=================
+*****************
 
 pyf.componentized permits to build data networks described in xml files, embeding python code or references to plugins.
 
 For a quick overview, see the :ref:`componentized tutorial <componentizedtutorial>`.
 
 Public API
-----------
+==========
 
 .. toctree::
    :maxdepth: 2
 
 
 Built-In plugins
-----------------
+================
 
 .. toctree::
    :maxdepth: 2

documentation/source/contents/modules/core/dataflow/tutorial.rst

+.. _lowleveltutorial:
+
 Low-level tutorial
 ------------------
 

documentation/source/contents/modules/core/index.rst

+****************
 PyF core modules
-================
+****************
 
 The core PyF modules are organised in seven packages.
 
 .. module:: pyf.dataflow
 
 :mod:`pyf.dataflow` Dataflow Core
----------------------------------
+=================================
 
 This is the core package for flow-based programming, it contains the building blocks for pyf.manager, pyf.componentized and pyf.services.
 
 .. module:: pyf.manager
 
 :mod:`pyf.manager` Data Manager
--------------------------------
+===============================
 
 It contains managers to handle flow networks or paths.
 
 .. module:: pyf.splitter
 
 :mod:`pyf.splitter` Low-Level Data Splitter
--------------------------------------------
+===========================================
 
 This is a low-level utility library used to split input flows and store them in buckets on the filesystem.
 

documentation/source/contents/modules/core/manager/tutorial.rst

+.. _networktutorial:
+
 Network tutorial
 ----------------
 

documentation/source/contents/modules/index.rst

+###########
 PyF Modules
-===========
+###########
 
 .. toctree::
    :maxdepth: 3

documentation/source/contents/modules/services/index.rst

 .. module:: pyf.services
 
+************
 PyF.Services
-============
+************
 
 PyF Services is a layer using all the other parts of PyF to provide a web service
 around data management, events, and data flow designing and scheduling.
    tutorial
 
 Features
---------
+========
+
+This section is a work in progress, sorry.
 
 .. toctree::
    :maxdepth: 2
    features/scheduling
 
 Data Model
-----------
+==========
+
+This section is a work in progress, sorry.
 
 .. toctree::
    :maxdepth: 2
    datamodel/tubelayers
 
 HTTP Connector
---------------
+==============
 
 You can manage your PyF Services instance via an http connector using the Rest API.
 
+This section is a work in progress, sorry.
+
 .. toctree::
    :maxdepth: 2
    

documentation/source/contents/modules/services/tutorial.rst

+.. _designtutorial:
+
+Tube Design Tutorial
+--------------------
+
+This tutorial assumes you've already `installed <http://pyfproject.org/en/getting-started>`_ and `setup <http://pyfproject.org/en/getting-started/configuring>`_ pyf.services.
+If you haven't, please follow the steps on the `getting started <http://pyfproject.org/en/getting-started>`_ pages of the website.
+
+This tutorial will show how to design a simple tube directly with your web browser on your PyF.Services instance.
+
+The tube that we will create is similar to the ones we created in the other tutorials (one producer, two filters, two adapters and one csv writer, similar to a process managing user accounts for security).
+
+Finding you way around and creating the tube
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+#. Click on the Tube menu item.
+     .. image:: ../../../_static/tube_menu.png
+#. Now, create a new tube in design view
+     .. image:: ../../../_static/newtube.png
+#. Name your tube, declare it as active and standalone
+     .. image:: ../../../_static/props.png
+
+Designing the tube
+^^^^^^^^^^^^^^^^^^
+
+1. First, let's drag and drop a code producer on our tube.
+     Drag and drop from the left list:
+       .. image:: ../../../_static/dragndrop-anim.png
+2. Our node is pretty empty right now.
+  a. First, we will name it: enter "source1" in the "Name" field.
+  b. Then, enter code in the code editor.
+     We will add a generator that will yield simple python objects representing users:
+       .. code-block:: python
+
+            class User(object):
+                def __init__(self, name, email, level):
+                    self.name = name
+                    self.email = email
+                    self.level = level
+            
+            def get_source():
+                for index in range(0, 10):
+                    yield User(
+                            "John%04d" % index,
+                            "john%04d@some-where.com" % index,
+                            ['high', 'low', 'high', 'high', 'low'][index%5])
+     .. HINT::
+          To get more information about nodes from code (code items), see :class:`pyf.componentized.components.CodeComponent`
+  c. Here is what your node should look like:
+       .. image:: ../../../_static/producer.png
+3. Now, let's separate the flow in two branches, one with the "high" level and another branch with the "low" and "medium" items.
+  a. For our first branch, we will use a plugin, named :class:`SimpleFilter <pyf.components.adapters.standardtools.SimpleFilter>` that will only let the items validating the expression "item.level == 'high'" pass through it.
+      .. image:: ../../../_static/filter1.png
+  b. Link the OUT port of the producer to the IN port of the filter adapter.
+      .. image:: ../../../_static/link-dnd-small.png
+  c. For our other branch, let's try another method of filtering : a custom code component.
+     
+     .. NOTE:: 
+        A very important thing when you make adapters, is to remember the way PyF works : it uses generators.
+        
+        So, to let the other branches keep executing you have to yield items. For that matter, we use a special Python objects : Ellipsis.
+        Those objects will be ignored by the rest of the components but let you say "I've received an item, but I don't want to send anything in the flow" (it is very useful for filters or aggregators that will take a lot of items and yield Totals at the end for example).
+     
+     A good example code for custom filtering is this:
+        .. code-block:: python
+
+            def low_or_med(items):
+                for item in items:
+                    if item.level == "low" or item.level == "med":
+                        yield item
+                    else:
+                        yield Ellipsis
+  d. Now we have our two branches:
+      .. image:: ../../../_static/two_branches.png
+4. Now that we have two branches, we want to adapt data in those branches.
+   There is a few ways to do that, and like at the step before, we will try two ways:
+  a. For our first branch, we will use the plugin way, with a plugin named :class:`ComputeAttributes<pyf.components.adapters.standardtools.ComputeAttributes>` that is specialised in adaptation of input objects by defining attribute getters on them, without modifying the input object (the objects that are outputed by this plugin are proxy objects, :class:`DynamicObjectAdapter <pyf.components.adapters.standardtools.DynamicObjectAdapter>` with attribute getters).
+      .. image:: ../../../_static/adapter1.png
+     We added an adapter named numeric_level that translates the level in a numeric fashion.
+      .. NOTE::
+         Each of the attribute getters can access item (the adapted item, with access to other attributes getter defined, and base_item, the item not adapted (useful to redefine attributes)).
+  b. For our second branch, we will do a code version. 
+     Here is the code we use for this:
+        .. code-block:: python
+
+            def set_numeric_level(items):
+                for item in items:
+                    item.numeric_level = 0
+                    yield item
+     Please not that this example isn't the best one as we modify the entering object (so you shouldn't make it pass in another branch at the same time), using Packets removes this problem as packets ca be modified and won't be the same on all branches (another solution is to use DynamicObjectAdapter).
+5. Last step is to add a CSVWriter. This is self explanatory, really. Just one thing, if you don't specify the attribute, the column title will be used.
+
+Here is what your tube should look like :
+   .. image:: ../../../_static/tutorial_tube.png
+
+6. Oh, last but not least, you should "Save" your Tube (there is a button for that on the left).
+
+Launching your tube
+^^^^^^^^^^^^^^^^^^^
+
+1. To launch your tube, either click the launch button from the design view or visit the tube page (the green arrow on the actions for the tubes in the tube list) and click Launch.
+2. Once launched, you are on the event track page with progression and messages.
+  - To add messages there, you can call message_callback in your nodes
+  - To update progression, use progression_callback in the producer.

documentation/source/contents/plugins/index.rst

+#######
 Plugins
-=======
+#######
 
+***********
 Information
------------
+***********
 
+****************
 Official plugins
-----------------
+****************
 
 Those plugins are made by the PyF team and can be found on the same repositories
 than pyf.componentized and pyf.services.
 
 Producers
-^^^^^^^^^
+=========
 
 .. toctree::
    :maxdepth: 2
    producers/webextractor
 
 Adapters
-^^^^^^^^
+========
 
 .. toctree::
    :maxdepth: 2
    adapters/standardtools
 
 Consumers
-^^^^^^^^^
+=========
 
 .. toctree::
    :maxdepth: 2
    consumers/xlsxwriter
    consumers/xmlwriter
 
+*******************
 Third-Party plugins
-^^^^^^^^^^^^^^^^^^^
+*******************
 
 Contribute your own plugins, and contact us to get listed here !

documentation/source/contents/tutorials.rst

 
 In the tutorials series we will cover the 4 main levels at which you can use PyF. Depending on which kind of way you want to use pyf you should read the corresponding tutorial.
 
-  - The low-level tutorial is written for people who want to understand the internals of pyf, but keep in mind that pyf's interest lies in its network and components architecture. Each tutorial will build upon the previous one and thus will continue the source code and exemples of the previous tuturial assuming that you know it already.
+  - The :ref:`low-level tutorial <lowleveltutorial>` is written for people who want to understand the internals of pyf, but keep in mind that pyf's interest lies in its network and components architecture. Each tutorial will build upon the previous one and thus will continue the source code and exemples of the previous tuturial assuming that you know it already.
 
-  - The network tutorial shows how to wire components together using a network manager instead of building all the wires between components by yourself.
+  - The :ref:`network tutorial <networktutorial>` shows how to wire components together using a network manager instead of building all the wires between components by yourself.
 
-  - The componentized (plugin-based) layer tutorial shows how to wire components without using code.
+  - The :ref:`componentized (plugin-based) layer tutorial <componentizedtutorial>` shows how to wire components without using code.
 
-  - Finally, the design tool of pyf.services (the web service) shows how to use the componentized layer without writing xml code, focusing on what you want to achieve.
+  - Finally, the :ref:`design tool <designtutorial>` of pyf.services (the web service) shows how to use the componentized layer without writing xml code, focusing on what you want to achieve.
 
-If you want to go straight to the fun part, I recommend reading directly the design tutorial, and then, to understand how it works at lower levels reading the others tutorial if needed.
+If you want to go straight to the fun part, I recommend reading directly the :ref:`design tutorial <designtutorial>`, and then, to understand how it works at lower levels reading the others tutorial if needed.
 
 In the tutorials we consider that you have successfully installed the necessary pyf layers. If you haven't, please go to the corresponding section on the website.
 
    modules/core/dataflow/tutorial
    modules/core/manager/tutorial
    modules/componentized/tutorial
+   modules/services/tutorial
+   
 
 
-

documentation/source/index.rst

    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
+###########################
 PyF Framework Documentation
-===========================
+###########################
 
+============
 Introduction
-------------
+============
 
 PyF is a library for the python language that enables you to use flow-based programming.
 
 
     PyF is architectured with different layers, each level is based on the level directly under it. :ref:`Read more ...  <architecture>`
 
+========
 Contents
---------
+========
 
 .. toctree::
    :maxdepth: 3
    contents/modules/index
    contents/plugins/index
 
-**Indices and tables** :
-
-    * :ref:`genindex`
-    * :ref:`modindex`
-    * :ref:`search`
-
+.. **Indices and tables** :
+..
+..    * :ref:`genindex`
+..    * :ref:`modindex`
+..    * :ref:`search`