Commits

Maciej Fijalkowski  committed 48522e7

(fijal, arigo) in the typical pypy style copy-paste beamerdefs

  • Participants
  • Parent commits e29ffc1

Comments (0)

Files changed (6)

File talk/pycon2013/pypy_without_gil/Makefile

+# you can find rst2beamer.py and inkscapeslide.py here:
+# http://bitbucket.org/antocuni/env/src/619f486c4fad/bin/rst2beamer.py
+# http://bitbucket.org/antocuni/env/src/619f486c4fad/bin/inkscapeslide.py
+
+
+talk.pdf: talk.rst author.latex title.latex stylesheet.latex
+	rst2beamer --stylesheet=stylesheet.latex --documentoptions=14pt talk.rst talk.latex || exit
+	sed 's/\\date{}/\\input{author.latex}/' -i talk.latex || exit
+	#sed 's/\\maketitle/\\input{title.latex}/' -i talk.latex || exit
+	pdflatex talk.latex  || exit
+
+view: talk.pdf
+	evince talk.pdf &
+
+xpdf: talk.pdf
+	xpdf talk.pdf &

File talk/pycon2013/pypy_without_gil/author.latex

+\definecolor{rrblitbackground}{rgb}{0.0, 0.0, 0.0}
+
+\title[Python without the GIL]{Python without the GIL}
+\author[arigo, fijal]
+{Armin Rigo \\ Maciej Fijałkowski}
+
+\institute{Pycon US 2013}
+\date{March 15 2013}

File talk/pycon2013/pypy_without_gil/beamerdefs.txt

+.. colors
+.. ===========================
+
+.. role:: green
+.. role:: red
+
+
+.. general useful commands
+.. ===========================
+
+.. |pause| raw:: latex
+
+   \pause
+
+.. |small| raw:: latex
+
+   {\small
+
+.. |end_small| raw:: latex
+
+   }
+
+.. |scriptsize| raw:: latex
+
+   {\scriptsize
+
+.. |end_scriptsize| raw:: latex
+
+   }
+
+.. |strike<| raw:: latex
+
+   \sout{
+
+.. closed bracket
+.. ===========================
+
+.. |>| raw:: latex
+
+   }
+
+
+.. example block
+.. ===========================
+
+.. |example<| raw:: latex
+
+   \begin{exampleblock}{
+
+
+.. |end_example| raw:: latex
+
+   \end{exampleblock}
+
+
+
+.. alert block
+.. ===========================
+
+.. |alert<| raw:: latex
+
+   \begin{alertblock}{
+
+
+.. |end_alert| raw:: latex
+
+   \end{alertblock}
+
+
+
+.. columns
+.. ===========================
+
+.. |column1| raw:: latex
+
+   \begin{columns}
+      \begin{column}{0.45\textwidth}
+
+.. |column2| raw:: latex
+
+      \end{column}
+      \begin{column}{0.45\textwidth}
+
+
+.. |end_columns| raw:: latex
+
+      \end{column}
+   \end{columns}
+
+
+
+.. |snake| image:: ../../img/py-web-new.png
+           :scale: 15%
+           
+
+
+.. nested blocks
+.. ===========================
+
+.. |nested| raw:: latex
+
+   \begin{columns}
+      \begin{column}{0.85\textwidth}
+
+.. |end_nested| raw:: latex
+
+      \end{column}
+   \end{columns}

File talk/pycon2013/pypy_without_gil/pypy_without_gil.txt

 * needs careful usage of locks
 
 * mostly hindered by the GIL in CPython
-  (but not in Jython)
+  (but not in Jython or IronPython)
 
 
 Classic solutions
 
 * no shared mutable state at all
 
-* copying, keeping in sync are your problem
+* copying, keeping in sync is your problem
 
-* memory usage is often multiplied (unless you're lucky with ``fork``)
+* serializing and deserializing is expensive and hard
+
+* memory usage is often multiplied (unless you're lucky with ``fork``,
+  but not on Python)
 
 
 Classic solutions
 
 * MPI: message passing, with limited shared state
 
-* etc.: tons of experiments that never caught on
+* etc.: tons of experiments that never caught on in the mainstream
 
 
 Classic solutions
 
 
 Demo
----------------------------
+----
 
 * ``pypy-stm``
 
 
 
 Demo
----------------------------
+----
 
 - demo: 
 
 
   * transactions2.py
 
+Status of the implementation
+----------------------------
 
+* mostly works
 
-- missing in pypy-stm:
+* major GC collections missing (leaks slowly)
 
-  * GC major collection
-  * JIT
-  * some more optimizations possible (= tons of)
+* JIT integration is not done
+
+* tons of optimizations possible
+
+
 
 
 - __pypy__.thread.atomic

File talk/pycon2013/pypy_without_gil/stylesheet.latex

+\usetheme{Boadilla}
+\usecolortheme{whale}
+\setbeamercovered{transparent}
+\setbeamertemplate{navigation symbols}{}
+
+\definecolor{darkgreen}{rgb}{0, 0.5, 0.0}
+\newcommand{\docutilsrolegreen}[1]{\color{darkgreen}#1\normalcolor}
+\newcommand{\docutilsrolered}[1]{\color{red}#1\normalcolor}
+
+\newcommand{\green}[1]{\color{darkgreen}#1\normalcolor}
+\newcommand{\red}[1]{\color{red}#1\normalcolor}

File talk/pycon2013/pypy_without_gil/talk.rst

+.. include:: beamerdefs.txt
+
+====================
+PyPy without the GIL
+====================
+
+Intro
+---------------------------
+
+- PyPy intro
+
+- thanks to contributors for STM
+
+
+Problem
+-------
+
+* An existing complex, large program that does stuff
+
+* "stuff" consists of bits that are mostly independent from each other
+
+|pause|
+
+* ... but not quite
+
+
+Problem
+-------
+
+* We want to parallelize the program to use all these cores
+
+* We have some shared mutable state
+
+|pause|
+
+* Not too much of it --- otherwise, no chance for parallelism
+
+* But still some
+
+
+Classic solutions
+-----------------
+
+Bare-metal multi-threading:
+
+* large shared state
+
+* needs careful usage of locks
+
+* mostly hindered by the GIL in CPython
+  (but not in Jython or IronPython)
+
+
+Classic solutions
+-----------------
+
+Multi-processing:
+
+* no shared mutable state at all
+
+* copying, keeping in sync is your problem
+
+* serializing and deserializing is expensive and hard
+
+* memory usage is often multiplied (unless you're lucky with ``fork``,
+  but not on Python)
+
+
+Classic solutions
+-----------------
+
+A range of intermediate solutions:
+
+* MPI: message passing, with limited shared state
+
+* etc.: tons of experiments that never caught on in the mainstream
+
+
+Classic solutions
+-----------------
+
+The typical solution for web servers:
+
+* run independent processes
+
+* share data only via the database
+
+* the database itself handles concurrency with transactions
+
+
+Demo
+----
+
+* ``pypy-stm``
+
+* internally based on "transactions" (STM, HTM)
+
+
+Demo
+----
+
+- demo: 
+
+  * multithreading.py
+
+  * multithreading2.py
+
+  * message_passing.py
+
+  * transactions2.py
+
+Status of the implementation
+----------------------------
+
+* mostly works
+
+* major GC collections missing (leaks slowly)
+
+* JIT integration is not done
+
+* tons of optimizations possible
+
+
+
+
+- __pypy__.thread.atomic
+
+  * with atomic:
+        print "hello", username
+
+  * the illusion of serialization
+
+
+Lower level: STM
+---------------------------
+
+- pypy and not cpython?
+
+
+- STM = Software Transactional Memory
+
+
+- Basic idea: each thread runs in parallel, but everything it does is
+  in a series of "transactions"
+
+
+- A transaction keeps all changes to pre-existing memory "local"
+
+
+- The changes are made visible only when the transaction "commits"
+
+
+- The transaction will "abort" if a conflict is detected, and
+  it will be transparently retried
+
+
+- Non-reversible operations like I/O turn the transaction "inevitable"
+  and stop progress in the other threads
+
+
+- __pypy__.thread.last_abort_info() -> traceback-like information
+
+
+- (GC-supported structure: when we want to modify a pre-existing object,
+  we first copy it into "local" memory, and when we commit, it becomes
+  the newer version of the old pre-existing object; we end up with a
+  chained list of versions, and we have to make sure we always use the
+  latest one.  We rely on GC major collections to free them eventually.)
+
+
+- alternative: HTM...?
+
+
+Higher level: not threads
+---------------------------
+
+
+- xxx memory usage good
+
+
+- based on (and fully compatible with) threads
+
+  * existing multithreaded programs work
+
+
+- but opens up unexpected alternatives
+
+  * we can run multiple threads but at the same time use ``atomic``
+
+  * with the GIL-based idea of ``atomic`` it doesn't make sense:
+    we have multiple threads but they're all using ``atomic``, i.e.
+    only one at a time will ever run...  except no :-)
+
+
+- transaction.py
+
+  * demo
+
+  * illusion of serial execution: can "sanely" reason about algorithms
+
+  * "Conflict tracebacks"
+
+  * Might need to debug them --- but they are performance bugs, not
+    correctness bugs
+
+  * The idea is that we fix only XX% of the bugs and we are done
+
+  * Maybe some new "performance debugger" tools might help too
+
+
+- TigerQuoll, 1st March: same idea with JavaScript (for servers)
+
+  * various models possible:
+
+    . events dispatchers
+
+    . futures
+
+    . map/reduce, scatter/gather
+
+
+- Event dispatchers
+
+  * twisted, tulip, etc.
+
+  * run the event dispatcher in one thread (e.g. the main thread),
+    and schedule the actual events to run on a different thread from
+    a pool
+
+  * the events are run with ``atomic``, so that they appear to run
+    serially
+
+  * does not rely on any change to the user program