Commits

Jan-Philip Gehrcke committed 27b8d33

Doc/readme cleanup.

Comments (0)

Files changed (2)

       hosted at Bitbucket.
 
 
-Is gipc reliable?
-=================
-Development of gipc began in late 2012, so it is still not being mature.
-However, as of version 0.3, I am not aware of severe issues. To my
-knowledge, gipc has already been deployed in serious projects. Generally, you
-should be aware of the fact that mixing any of fork, threads, greenlets and an
-event loop library such as libev bears the potential for various kinds of
-corner-case disasters. One could argue that ``fork()`` in the
-context of libev without doing a clean ``exec`` in the child already *is*
-broken design. However, many people would like to do exactly this and gipc's
-basic approach has proven to work in such cases. gipc is developed with a
-strong focus on reliability and with best intentions in mind,
-and via unit testing, gipc has been validated to work reliably in scenarios of
-low and medium complexity. Of course, gipc cannot rescue an a priori ill-posed
-approach. Now it is up to you to evaluate gipc in the context of your project
--- please let me know how gipc performs for you.
-
-
-Where should I download gipc?
-=============================
+Where to download gipc?
+=======================
     - Releases: `PyPI <http://pypi.python.org/pypi/gipc>`_.
     - Development version: `Hg repository <https://bitbucket.org/jgehrcke/gipc>`_.
 
         - :ref:`What is gipc good for? <what>`
         - :ref:`Usage <usage>`
         - :ref:`Technical notes <technotes>`
+        - :ref:`Is gipc reliable? <reliable>`
         - :ref:`Code, requirements, download, installation <installation>`
         - :ref:`Notes for Windows users <winnotes>`
         - :ref:`Author, license, contact <contact>`
 There is plenty of motivation for using multiple processes in event-driven
 architectures. The assumption behind gipc is that applying multiple processes
 that communicate among each other (whereas each process has its own event loop)
-can be a decent solution for many types of problems in such architectures.
-First of all, it helps decoupling system components by making each process
-responsible for one part of the architecture only. Furthermore, even a
-generally I/O-limited application might at some point -- if executed on one CPU
-core only -- become bound to the performance of this CPU core. In these cases,
-the distribution of tasks among multiple processes is an efficient way to make
-use of multi-core machines and easily increases the application's performance.
+can be a decent solution for many types of problems. First of all, it helps
+decoupling system components by making each process responsible for one part of
+the architecture only. Furthermore, even a generally I/O-intense application
+might at some point become CPU bound. In these cases, the distribution of tasks
+among multiple processes is an efficient way to make use of multi-core machines
+and easily increases the application's performance.
 
 However, canonical usage of Python's multiprocessing module within a
 gevent-powered application may raise various problems and most likely breaks
             readchild.join()
 
 Although quite simple, this code would have various negative side-effects if
-used with the canonical multiprocessing API instead of
-``gipc.start_process()`` and ``gipc.pipe()``, as outlined in the next paragraph.
+used with the canonical multiprocessing API instead of ``gipc.start_process()``
+and ``gipc.pipe()``, as outlined in the next paragraph.
 
 
 What are the challenges and what is gipc's approach?
 
 In addition, the code above contains several non-cooperatively blocking method
 calls: ``readchild.join()`` as well as the ``send()``/``recv()`` calls (of
-``multiprocessing.Connection`` objects in general) block the calling thread and
-do not allow for context switches.
+``multiprocessing.Connection`` objects in general) block the calling greenlet
+non-cooperatively and do not allow for context switches into other greenlets.
 
 **Solution:**
 
 libev event loop and create their own fresh versions of these objects. This
 way, inherited greenlets as well as libev watchers become orphaned -- the fresh
 hub and event loop are not connected to them anymore. Consequently, execution
-of code related to these inherited greenlets and libev watchers is efficiently
+of code related to these inherited greenlets and watchers is efficiently
 prevented without the need to deactivate or kill them one by one.
 
 Furthermore, on POSIX-compliant systems, gipc entirely avoids multiprocessing's
 child monitoring implementation (which is based on ``wait``) and instead uses
-libev's wonderful child watcher system (based on SIGCHLD signal transmission).
+libev's wonderful child watcher system (based on SIGCHLD signal transmission),
+enabling gevent-cooperative waiting for child termination.
 
-For the sake of gevent-cooperative inter-process communication, gipc provides
-efficient pipe-based data transport channels. Of course, gipc takes care of
-closing dispensable pipe handles (file descriptors) in the parent as well as in
-the child after forking.
+For the sake of gevent-cooperative inter-process communication, gipc uses
+efficient pipe-based data transport channels with non-blocking I/O. Of course,
+gipc takes care of closing dispensable pipe handles (file descriptors) in the
+parent as well as in the child after forking.
 
 Overall, gipc main goal is to allow for the integration of child processes in
 your gevent-powered application via a simple API -- on POSIX-compliant systems
   objects. Reading and writing on pipes is done with ``gevent``'s cooperative
   versions of ``os.read()`` and ``os.write()`` (on POSIX-compliant systems they
   use non-blocking I/O, on Windows a threadpool is used). On Linux, my test
-  system (Xeon E5630) achieved a payload transfer rate of 1200 MB/s and a message
-  transmission rate of 100.000 messages/s through one pipe between two
+  system (Xeon E5630) achieved a payload transfer rate of 1200 MB/s and a
+  message transmission rate of 100.000 messages/s through one pipe between two
   processes.
 
 - Child process creation and invocation is done via a thin wrapper around
   gipc + gevent fails for you.
 
 
+.. _reliable:
+
+Is gipc reliable?
+=================
+Development of gipc began in late 2012, so it is still not being mature.
+However, as of version 0.3, I am not aware of severe issues. To my
+knowledge, gipc has already been deployed in serious projects. Generally, you
+should be aware of the fact that mixing any of fork, threads, greenlets and an
+event loop library such as libev bears the potential for various kinds of
+corner-case disasters. One could argue that ``fork()`` in the
+context of libev without doing a clean ``exec`` in the child already *is*
+broken design. However, many people would like to do exactly this and gipc's
+basic approach has proven to work in such cases. gipc is developed with a
+strong focus on reliability and with best intentions in mind,
+and via unit testing, it has been validated to work reliably in scenarios of
+low and medium complexity. Of course, gipc cannot rescue an a priori ill-posed
+approach. Now it is up to you to evaluate gipc in the context of your project
+-- please let me know how it performs for you.
+
+
 .. _installation:
 
 Code, requirements, download, installation
 ==========================================
 
-Code
-----
 gipc's Mercurial repository is hosted at
-`Bitbucket <https://bitbucket.org/jgehrcke/gipc>`_. It also contains a
-changelog and license information.
+`Bitbucket <https://bitbucket.org/jgehrcke/gipc>`_.
 
-
-Requirements
-------------
+gipc requires:
 
 - `gevent <http://gevent.org>`_ >= 1.0 (currently, gipc is tested against
   gevent 1.0). Download recent gevent releases
   `here <https://github.com/surfly/gevent/downloads>`_ or from
   `PyPI <https://pypi.python.org/pypi/gevent>`_.
-- The unit tests are ensured to pass on CPython 2.6 and 2.7 on Linux as well
-  as on Windows.
+- CPython 2.6 or 2.7 (Python 3 will be supported as soon as gevent supports
+  it). Unit tests are made sure to work on Linux as well as Windows.
 
-
-Download & install via pip
---------------------------
-
-The latest official gipc release from PyPI can be pulled and installed via
+The latest gipc release from PyPI can be pulled and installed via
 `pip <http://www.pip-installer.org>`_::
 
     $ pip install gipc
 
     $ pip install hg+https://bitbucket.org/jgehrcke/gipc
 
-Note that the latter requires a recent version of
-`distribute <http://packages.python.org/distribute/>`_ which can be installed
-by executing
-`distribute_setup.py <http://python-distribute.org/distribute_setup.py>`_.
-
-pip is recommended over easy_install. pip installation instructions can be
-found `here <http://www.pip-installer.org/en/latest/installing.html>`_.
-
-
-Install directly via setup.py
------------------------------
-
-Download and extract the latest gipc release archive from
-`PyPI <http://pypi.python.org/pypi/gipc/>`_. Extract it and invoke::
-
-    $ python setup.py install
-
-The same can be done with the latest development version of gipc which
-can be downloaded from `Bitbucket <https://bitbucket.org/jgehrcke/gipc>`_.
-
 
 .. _winnotes:
 
 gipc.pipe()-based messaging from greenlet in parent to child
 ============================================================
 
-Some basic concepts are explained by means of this simple messaging example:
+Very basic gevent and gipc concepts are explained by means of this simple
+messaging example:
 
 .. code::
 
         main()
 
 The context manager ``with gipc.pipe() as (r, w)`` creates a pipe with read
-handle ``r`` and write handle ``w``. On context exit (latest) the pipe ends will
-be closed properly.
+handle ``r`` and write handle ``w``. On context exit (latest) the pipe ends
+will be closed properly.
 
-Within the context, a child process is spawned via ``gipc.start_process()``.
-The read handle ``r`` is provided to the child process. The child invokes
-``child_process(r)`` where an endless loop waits for objects on the read end of
-the pipe. Upon retrieval, it immediately writes them to stdout.
+After creating the pipe context, the above code spawns a child process via
+``gipc.start_process()``. The child process is instructed to execute the target
+function named ``child_process`` whereas the pipe read handle ``r`` is provided
+as an argument to this target function. Within ``child_process()`` an endless
+loop waits for objects on the read end of the pipe via the cooperatively
+blocking call to ``reader.get()``. Upon retrieval, it immediately writes their
+string representation to stdout.
 
-While child process ``p`` is running, a greenlet ``wg`` is started in the main
-process. It executes the function ``writegreenlet`` and passes the write handle
-``w`` as an argument. Within this greenlet, one string per second is written to
-the write end of the pipe.
+After invocation of the the child process (represented by an object bound to
+name ``p``), a greenlet ``wg`` is spawned within the main process. This
+greenlet executes the function ``writegreenlet``, whereas the pipe write handle
+``w`` is provided as an argument. Within this greenlet, one string per second
+is written to the write end of the pipe.
 
-After spawning ``wg``, ``p.join()`` is called immediately, i.e. the write
-greenlet is running while ``p.join()`` waits for the child process to terminate.
-In this state, one message per second is passed between parent and child until a
-``KeyboardInterrupt`` exception is raised in the parent.
+After spawning ``wg``, ``p.join()`` is called immediately in the parent
+process. ``p.join()`` is blocking cooperatively, i.e. it allows for a context
+switch into the write greenlet (this actually is the magic behind
+gevent/greenlets). Hence, the write greenlet is 'running' while ``p.join()``
+cooperatively waits for the child process to terminate. The write greenlet
+spends most of its time in ``gevent.sleep()``, which is also blocking
+cooperatively, allowing for context switches back to the main greenlet in the
+parent process, which is executing ``p.join()``. In this state, one message per
+second is passed between parent and child until a ``KeyboardInterrupt``
+exception is raised in the parent.
 
 Upon ``KeyboardInterrupt``, the parent first kills the write greenlet and blocks
 cooperatively until it has stopped. Then it terminates the child process (via
 Serving multiple clients (in child) from one server (in parent)
 ===============================================================
 
-For pure API and reliability demonstration purposes, this example implements TCP
-communication between a server in the parent process and multiple clients in
-one child process:
+For pure API and reliability demonstration purposes, this example implements
+TCP communication between a server in the parent process and multiple clients
+in one child process:
 
 1)  gevent's ``StreamServer`` is started in a greenlet within the initial
     (parent) process. For each connecting client, it receives one