1. Pypy
  2. Untitled project
  3. pypy


holger krekel  committed bb0faaf

merge the documentation branch back to dist

  • Participants
  • Parent commits e6f80b3
  • Branches default

Comments (0)

Files changed (77)

File pypy/doc/__init__.py

View file

File pypy/doc/_ref.txt

View file
+.. _`demo/`: ../../demo
+.. _`lib-python/`: ../../lib-python
+.. _`annotation/`:
+.. _`pypy/annotation`: ../../pypy/annotation
+.. _`annotation/binaryop.py`: ../../pypy/annotation/binaryop.py
+.. _`doc/`: ../../pypy/doc
+.. _`doc/revreport/`: ../../pypy/doc/revreport
+.. _`interpreter/`:
+.. _`pypy/interpreter`: ../../pypy/interpreter
+.. _`pypy/interpreter/argument.py`: ../../pypy/interpreter/argument.py
+.. _`pypy/interpreter/function.py`: ../../pypy/interpreter/function.py
+.. _`pypy/interpreter/gateway.py`: ../../pypy/interpreter/gateway.py
+.. _`pypy/interpreter/generator.py`: ../../pypy/interpreter/generator.py
+.. _`pypy/interpreter/mixedmodule.py`: ../../pypy/interpreter/mixedmodule.py
+.. _`pypy/interpreter/nestedscope.py`: ../../pypy/interpreter/nestedscope.py
+.. _`pypy/interpreter/pyopcode.py`: ../../pypy/interpreter/pyopcode.py
+.. _`pypy/interpreter/typedef.py`: ../../pypy/interpreter/typedef.py
+.. _`lib/`:
+.. _`pypy/lib/`: ../../pypy/lib
+.. _`lib/test2/`:
+.. _`pypy/lib/test2`: ../../pypy/lib/test2
+.. _`module/`:
+.. _`pypy/module`: ../../pypy/module
+.. _`module/__builtin__/`: ../../pypy/module/__builtin__
+.. _`pypy/module/__builtin__/__init__.py`: ../../pypy/module/__builtin__/__init__.py
+.. _`module/_sre_pypy/`: ../../pypy/module/_sre_pypy
+.. _`module/parser/`: ../../pypy/module/parser
+.. _`module/recparser/`: ../../pypy/module/recparser
+.. _`module/sys/`: ../../pypy/module/sys
+.. _`objspace/`:
+.. _`pypy/objspace`: ../../pypy/objspace
+.. _`objspace/flow/`: ../../pypy/objspace/flow
+.. _`objspace/std/`:
+.. _`pypy/objspace/std`: ../../pypy/objspace/std
+.. _`objspace/thunk.py`: ../../pypy/objspace/thunk.py
+.. _`objspace/trace.py`:
+.. _`pypy/objspace/trace.py`: ../../pypy/objspace/trace.py
+.. _`rpython/`: ../../pypy/rpython
+.. _`rpython/lltype.py`: ../../pypy/rpython/lltype.py
+.. _`rpython/rint.py`: ../../pypy/rpython/rint.py
+.. _`rpython/rlist.py`: ../../pypy/rpython/rlist.py
+.. _`rpython/rmodel.py`: ../../pypy/rpython/rmodel.py
+.. _`pypy/test_all.py`: ../../pypy/test_all.py
+.. _`tool/`: ../../pypy/tool
+.. _`tool/pytest/`: ../../pypy/tool/pytest
+.. _`tool/tb_server/`: ../../pypy/tool/tb_server
+.. _`translator/`:
+.. _`pypy/translator`: ../../pypy/translator
+.. _`pypy/translator/annrpython.py`: ../../pypy/translator/annrpython.py
+.. _`translator/c/`: ../../pypy/translator/c
+.. _`translator/java/`: ../../pypy/translator/java
+.. _`translator/llvm2/`: ../../pypy/translator/llvm2
+.. _`translator/tool/`: ../../pypy/translator/tool

File pypy/doc/architecture.txt

View file
+PyPy - Architecture Overview 
+.. contents::
+.. sectnum::
+This document gives an overview of the goals and architecture of PyPy.
+See also `getting started`_ for a practical introduction.
+.. _`getting started`: getting-started.html
+Mission statement 
+PyPy is a reimplementation of Python_ written in Python itself, flexible and
+easy to experiment with.  Our long-term goals are to target a large variety of
+platforms, small and large, by providing a compiler toolsuite that can produce
+custom Python versions.  Platform, Memory and Threading models are to become
+aspects of the translation process - as opposed to encoding low level details
+into a language implementation itself.  Eventually, dynamic optimization techniques
+- implemented as another translation aspect - should become robust against
+language changes.
+PyPy - an implementation of Python in Python
+It has become a tradition in the development of computer languages to
+implement each language in itself. This serves many purposes. By doing so,
+you demonstrate the versatility of the language, and its applicability for
+large projects.  Writing compilers and interpreters are among the most 
+complex endeavours in software development. 
+An important aspect of implementing Python in Python is the high level of
+abstraction and compactness of the language. This allows an implementation
+that is, in some respects, easier to understand and play with than the one
+done in C.  Actually, the existing CPython implementation is mostly 
+well written and it is often possible to manually translate according
+CPython code to PyPy by just stripping away many low level details. 
+Another carrying idea in PyPy is to build the implementation in the form
+of a number of independent modules with clearly defined and well tested API's. 
+This eases reuse and allows experimenting with multiple implementations 
+of specific features.
+Later in the project, we will introduce optimizations, following the ideas
+of Psyco_ and Stackless_, that should make PyPy run Python programs 
+faster than CPython.
+.. _Python: http://www.python.org/doc/current/ref/ref.html
+.. _Psyco: http://psyco.sourceforge.net
+.. _Stackless: http://stackless.com 
+Status of the implementation (May 2005)
+In a number of one week sprints, attracting approximately 10 developers each,
+we made an almost complete implementation of Python in Python. Currently it is
+slow, benchmarking at a factor 2000 times slower than regular Python
+(henceforth referred to as CPython). This was expected because running a
+bytecode interpreter on top of CPython obviously imposes a
+double-interpretation penalty. 
+Our rather complete and Python 2.3-compliant interpreter is about 22000 lines 
+of code, with another 7000 lines of unit tests.  If we include the tools, the
+parts related to code analysis and generation, and the standard library
+modules ported from C, PyPy is now 55000 lines of code and 20000 lines of
+PyPy already passes a lot of CPython's own regression tests, including 90% of the 
+core tests not depending on C extension modules (most of the remaining 10% are
+arguably dependant on very obscure implementation details of CPython).
+For some time now the bleeding edge PyPy work has been focused on generating
+reasonably efficient C code from the source of PyPy, thereby reducing the
+speed penalty.  This goal is no longer far away.
+Higher level picture
+The various parts of PyPy have always been under more or less heavy
+refactoring since its inception. However, the higher level architecture
+remains rather simple and unchanged.  There are two independent basic
+The Standard Interpreter
+The *standard interpreter* is the subsystem implementing the Python language.
+It is divided in two components:
+- the `plain interpreter`_ which is responsible for interpreting 
+  code objects and implementing bytecodes,
+- the `standard object space`_ which implements creation, access and
+  modification of application level objects,
+Note that the *standard interpreter* can run fine on top of CPython 
+(the C Implementation of Python led by Guido van Rossum), if one is
+willing to pay for the double-interpretation performance penalty.
+Please note that we are using the term *interpreter* most often in
+reference to the *plain interpreter* which just knows enough to read,
+dispatch and implement *bytecodes* thus shuffling objects around on the
+stack and between namespaces.  The (plain) interpreter is completely
+ignorant of how to access, modify or construct objects and their
+structure and thus delegates such operations to a so called `Object Space`_.
+In addition, the standard interpreter requires a parser and bytecode compiler
+to turn the user's Python source code into a form amenable to
+interpretation.  This is currently still borrowed from CPython, but we
+have two experimental parser modules in the source tree which are in the
+process of being integrated.
+The Translation Process
+The *translation process* aims at producing a different (low-level)
+representation of our standard interpreter.  The *translation process* 
+is done in four steps:
+- producing a *flow graph* representation of the standard interpreter. 
+  A combination of a `plain interpreter`_ and a *flow object space*
+  performs *abstract interpretation* to record the flow of objects
+  and execution throughout a python program into such a *flow graph*.
+- the *annotator* which performs type inference on the flow graph 
+- the *typer* which, based on the type annotations, turns the flow graph
+  into one using only low-level, C-like operations
+- the *code generator* which translates the resulting flow graph into
+  another language, currently C or LLVM.
+See below for the `translation process in more details`_.
+.. _`plain interpreter`:
+The Bytecode Interpreter
+The *plain bytecode interpreter* handles python code objects. The interpreter can 
+build code objects from Python sources, when needed, by invoking Python's
+builtin compiler (we also have a way of constructing those code objects
+from python source only, but we have not integrated it yet).  Code objects
+are a nicely preprocessed, structured representation of source code, and
+their main content is *bytecode*.  In addition, code objects also know
+how to create a *frame* object which has the responsibility to
+*interpret* a code object's bytecode.  Each bytecode is implemented by a
+python function, which, in turn, delegates operations on
+application-level objects to an object space. 
+This part is implemented in the `interpreter/`_ directory.  People familiar
+with the CPython implementation of the above concepts will easily recognize
+them there.  The major differences are the overall usage of the `Object Space`_
+indirection to perform operations on objects, and the organization of the
+built-in modules (described `here`_).
+.. _`here`: coding-guide.html#modules
+.. _`objectspace`: 
+.. _`Object Space`: 
+The Object Space
+The object space creates all objects and knows how to perform operations
+on the objects. You may think of an object space as being a library
+offering a fixed API, a set of *operations*, with implementations that
+correspond to the known semantics of Python objects.  An example of an
+operation is *add*: add's implementations are, for example, responsible
+for performing numeric addition when add works on numbers, concatenation
+when add works on built-in sequences.
+All object-space operations take and return `application-level`_ objects.
+There are only a few, very simple, object-space operation which allows the
+interpreter to gain some knowledge about the value of an
+application-level object.
+The most important one is ``is_true()``, which returns a boolean
+interpreter-level value.  This is necessary to implement, for example,
+if-statements (or rather, to be pedantic, to implement the
+conditional-branching bytecodes into which if-statements get compiled). 
+We currently have four working object spaces which can be plugged into
+the interpreter:
+.. _`standard object space`:
+- The *Standard Object Space* is a complete implementation 
+  of the various built-in types and objects of Python.  The Standard Object
+  Space, together with the interpreter, is the foundation of our Python
+  implementation.  Internally, it is a set of `interpreter-level`_ classes
+  implementing the various `application-level`_ objects -- integers, strings,
+  lists, types, etc.  To draw a comparison with CPython, the Standard Object
+  Space provides the equivalent of the C structures ``PyIntObject``,
+  ``PyListObject``, etc.
+- the *Trace Object Space* wraps e.g. the standard 
+  object space in order to trace the execution of bytecodes, 
+  frames and object space operations.
+- the *Thunk Object Space* wraps another object space (e.g. the standard
+  one) and adds two capabilities: lazily computed objects (computed only when
+  an operation is performed on them), and "become", which completely and
+  globally replaces an object with another.
+- the *Flow Object Space* transforms a Python program into a
+  flow-graph representation, by recording all operations that the interpreter
+  would like to perform when it is shown the given Python program.  This
+  technique is explained `later in this document`_.
+For a description of the object spaces, please see the
+`objspace document`_.  The sources of PyPy contain the various object spaces
+in the directory `objspace/`_.
+.. _`objspace document`: objspace.html
+.. _`application-level`:
+.. _`interpreter-level`:
+Application-level and interpreter-level execution and objects
+Since Python is used for implementing all of our code base, there is a
+crucial distinction to be aware of: *interpreter-level* objects versus
+*application-level* objects.  The latter are the ones that you deal with
+when you write normal python programs.  Interpreter-level code, however,
+cannot invoke operations nor access attributes from application-level
+objects.  You will immediately recognize any interpreter level code in
+PyPy, because half the variable and object names start with a ``w_``, which
+indicates that they are `wrapped`_ application-level values. 
+Let's show the difference with a simple example.  To sum the contents of
+two variables ``a`` and ``b``, typical application-level code is ``a+b``
+-- in sharp contrast, typical interpreter-level code is ``space.add(w_a,
+w_b)``, where ``space`` is an instance of an object space, and ``w_a``
+and ``w_b`` are typical names for the wrapped versions of the two
+It helps to remember how CPython deals with the same issue: interpreter
+level code, in CPython, is written in C, and thus typical code for the
+addition is ``PyNumber_Add(p_a, p_b)`` where ``p_a`` and ``p_b`` are C
+variables of type ``PyObject*``. This is very similar to how we write
+our interpreter-level code in Python.
+Moreover, in PyPy we have to make a sharp distinction between
+interpreter- and application-level *exceptions*: application exceptions
+are always contained inside an instance of ``OperationError``.  This
+makes it easy to distinguish failures (or bugs) in our interpreter-level code
+from failures appearing in a python application level program that we are
+.. _`app-preferable`: 
+Application level is often preferable 
+Application-level code is substantially higher-level, and therefore
+correspondingly easier to write and debug.  For example, suppose we want
+to implement the ``update`` method of dict objects.  Programming at
+application level, we can write an obvious, simple implementation, one
+that looks like an **executable definition** of ``update``, for
+    def update(self, other):
+        for k in other.keys():
+            self[k] = other[k]
+If we had to code only at interpreter level, we would have to code
+something much lower-level and involved, say something like::
+    def update(space, w_self, w_other):
+        w_keys = space.call_method(w_other, 'keys')
+        w_iter = space.iter(w_keys)
+        while True:
+            try:
+                w_key = space.next(w_iter)
+            except OperationError, e:
+                if not e.match(space, space.w_StopIteration):
+                    raise       # re-raise other app-level exceptions
+                break
+            w_value = space.getitem(w_other, w_key)
+            space.setitem(w_self, w_key, w_value)
+This interpreter-level implementation looks much more similar to the C
+source code.  It is still more readable than it's C counterpart because 
+it doesn't contain memory management details and can use Python's native 
+exception mechanism. 
+In any case, it should be obvious that the application-level implementation 
+is definitely more readable, more elegant and more maintainable than the
+interpreter-level one.
+In fact, in almost all parts of PyPy, you find application level code in
+the middle of interpreter-level code.  Apart from some bootstrapping
+problems (application level functions need a certain initialization
+level of the object space before they can be executed), application
+level code is usually preferable.  We have an abstraction (called
+'Gateway') which allows the caller of a function to remain ignorant of
+whether a particular function is implemented at application or
+interpreter level. 
+.. _`wrapped`:
+The ``w_`` prefixes so lavishly used in the previous example indicate,
+by PyPy coding convention, that we are dealing with *wrapped* objects,
+that is, interpreter-level objects which the object space constructs
+to implement corresponding application-level objects.  Each object
+space supplies ``wrap`` and ``unwrap``, ``int_w``, ``interpclass_w``,
+etc. operations that move between the two levels for objects of simple
+built-in types; each object space also implements other Python types
+with suitable interpreter-level classes with some amount of internal
+For example, an application-level Python ``list``
+is implemented by the `standard object space`_ as an
+instance of ``W_ListObject``, which has an instance attribute
+``ob_item`` (an interpreter-level list which contains the
+application-level list's items as wrapped objects) and another attribute
+``ob_size`` which records the application-level list's length (we want
+to be able to do "over-allocation" in ``ob_item``, for the same reasons
+of performance that lead CPython to do it, and therefore the length of
+``ob_item`` is allowed to be greater than the length of the
+application-level list -- it is for this reason that the length in
+question has to be explicitly recorded in ``ob_size``).
+The rules are described in more details `in the coding guide`_.
+.. _`in the coding guide`: coding-guide.html#wrapping-rules
+.. _`translation process in more details`:
+.. _`later in this document`:
+.. _`initialization time`: 
+RPython, the Flow Object Space and translation
+One of PyPy's now-short-term objectives is to enable translation of our
+interpreter and standard object space into a lower-level language.  In
+order for our translation and type inference mechanisms to work
+effectively, we need to restrict the dynamism of our interpreter-level
+Python code at some point.  However, in the start-up phase, we are
+completely free to use all kind of nice python constructs, including
+metaclasses and execution of dynamically constructed strings.  However,
+when the initialization phase (mainly, the function
+``objspace.initialize()``) finishes, all code objects involved need to
+adhere to a more static subset of Python:
+Restricted Python, also known as `RPython`_. 
+The Flow Object Space then, with the help of our plain interpreter,
+works through those initialized RPython code objects.  The result of
+this `abstract interpretation`_ is a flow graph: yet another
+representation of a python program, but one which is suitable for
+applying translation and type inference techniques.  The nodes of the
+graph are basic blocks consisting of Object Space operations, flowing
+of values, and an exitswitch to one, two or multiple links which connect
+each basic block to other basic blocks. 
+The flow graphs are fed as input into the Annotator.  The Annotator,
+given entry point types, infers the types of values that flow through
+the program variables.  Here, the definition of `RPython`_ comes
+again into play: RPython code is restricted in such a way that the
+Annotator is able to infer consistent types.  In total, how much
+dynamism we allow in RPython depends, and is restricted by, the Flow
+Object Space and the Annotator implementation.  The more we can improve
+this translation phase, the more dynamism we can allow.  In some cases,
+however, it will probably be more feasible and practical to just get rid
+of some of the dynamism we use in our interpreter level code.  It is
+mainly because of this trade-off situation that the definition of
+RPython has been shifting quite a bit.  Although the Annotator is
+pretty stable now, and able to process the whole of PyPy, the RPython
+definition will probably continue to shift marginally as we improve it.
+The actual low-level code (and, in fact, also other high-level code) is
+emitted by "visiting" the type-annotated flow graph.  Currently we have
+a C-producing backend, and an LLVM-producing backend.  The former also
+accepts non-annotated or partially-annotated graphs, which allow us to
+test it on a larger class of programs than what the Annotator can (or
+ever will) fully process.
+A new piece of this puzzle, still being integrated (May 2005), is the
+*Typer*, which inputs the high-level types inferred by the Annotator and
+uses them to modify the flow graph in-place to replace its operations with
+low-level ones, directly manipulating C-like values and data structures.
+The complete translation process is described in more details in the
+`translation document`_.
+.. _`RPython`: coding-guide.html#rpython
+.. _`abstract interpretation`: theory.html#abstract-interpretation
+.. _`translation document`: translation.html
+.. include:: _ref.txt

File pypy/doc/coding-guide.txt

View file
+PyPy - Coding Guide
+.. contents::
+.. sectnum::
+This document describes coding requirements and conventions for
+working with the PyPy code base.  Please read it carefully and
+ask back any questions you might have.
+.. _`RPython`:
+Restricted Python
+We are writing a Python interpreter in Python, using Python's well known
+ability to step behind the algorithmic problems as language. At first glance,
+one might think this achieves nothing but a better understanding how the
+interpreter works.  This alone would make it worth doing, but we have much
+larger goals.
+CPython vs. PyPy
+Compared to the CPython implementation, Python takes the role of the C
+Code. We rewrite the CPython interpreter in Python itself.  We could
+also aim at writing a more flexible interpreter at C level but but we
+want to use Python to give an alternative description of the interpreter.
+The clear advantage is that such a description is shorter and simpler to
+read, and many implementation details vanish. The drawback of this approach is
+that this interpreter will be unbearably slow as long as it is run on top
+of CPython.
+To get to a useful interpreter again, we need to translate our
+high-level description of Python to a lower level one.  One rather
+straight-forward way is to do a whole program analysis of the PyPy
+interpreter and create a C source, again. There are many other ways,
+but let's stick with this somewhat canonical approach.
+our runtime interpreter is "restricted python"
+In order to make a C code generator feasible we restrict ourselves to a
+subset of the Python language, and we adhere to some rules which make
+translation to lower level languages more obvious.
+Unlike source-to-source translations (like e.g. Starkiller_) we start
+translation from live python code objects which constitute our Python
+interpreter.   When doing its work of interpreting bytecode our Python
+implementation must behave in a static way often referenced as
+.. _Starkiller: http://www.python.org/pycon/dc2004/papers/1/paper.pdf
+However, when the PyPy interpreter is started as a Python program, it
+can use all of the Python language until it reaches interpretation
+runtime. That is, during initialization our program is free to use the
+full dynamism of Python, including dynamic code generation.
+An example can be found in the current implementation which is quite
+elegant: For the definition of all the opcodes of the Python
+interpreter, the module ``dis`` is imported and used to initialize our
+bytecode interpreter.  (See ``__initclass__`` in `pyopcode.py`_).  This
+saves us from adding extra modules to PyPy. The import code is run at
+startup time, and we are allowed to use the CPython builtin import
+After the startup code is finished, all resulting objects, functions,
+code blocks etc. must adhere to certain runtime restrictions which we
+describe further below.  Here is some background for why this is so:
+during translation, a whole program analysis ("type inference") is
+performed, which makes use of the restrictions defined in RPython. This
+enables the code generator to emit efficient machine level replacements
+for pure integer objects, for instance.
+.. _`pyopcode.py`: http://codespeak.net/svn/pypy/dist/pypy/interpreter/pyopcode.py
+RPython Definition, not
+The list and exact details of the "RPython" restrictions are a somewhat
+evolving topic.  In particular, we have no formal language definition
+as we find it more practical to discuss and evolve the set of
+restrictions while working on the whole program analysis.  If you
+have any questions about the restrictions below then please feel
+free to mail us at pypy-dev at codespeak net.
+.. _`wrapped object`: coding-guide.html#wrapping-rules
+Flow restrictions
+  variables should contain values of at most one type as described in
+  `Object restrictions`_ at each control flow point, that means for
+  example that joining control paths using the same variable to
+  contain both a float and a int should be avoided. Mixing None
+  (basically with the role of a null pointer) and `wrapped objects`
+  and class instances is allowed.
+  all module globals are considered constants.
+**control structures**
+  all allowed
+  does not create an array. It is only allowed in for loops. The step argument
+  must be a constant.
+  run-time definition of classes or functions is not allowed.
+  generators are not supported.
++ fully supported
++ see below `Exception rules`_ for restrictions on exceptions raised by built-in operations
+Object restrictions
+We are using
+**integer, float, string, boolean**
+  avoid string methods and complex operations like slicing with a step
+  no variable-length tuples; use them to store or return pairs or n-tuples of
+  values. Each combination of types for elements and length constitute a separate
+  and not mixable type.
+  lists are used as an allocated array; list.append() does naive resizing, so as
+  far as possible use list comprehensions (see below).  list.extend() or the +=
+  operator are allowed and efficient.  Unless there is really a use case for it,
+  repetition is limited to initialization purposes: '[single_value] * length'.
+  dicts with string keys only (preferably the kind of strings that are usually
+  interned in CPython, i.e. short strings that look like identifiers).  The
+  implementation could safely decide that all dict keys should be interned.
+**list comprehensions**
+  may be used to create allocated, initialized array. the array size must be
+  computable in advance, which implies that we don't allow an if clause.
++ statically called functions may use defaults and a variable number of
+  arguments (which may be passed as a list instead of a tuple, so write code
+  that does not depend on it being a tuple).
++ dynamic dispatch enforces use of very simple signatures, equal for all
+  functions to be called in that context. At the moment, this occurs in the
+  opcode dispatch, only.
+**builtin functions**
+  A few builtin functions will be used, while this set is not defined
+  completely, yet. Some builtin functions are special forms:
++ may be used with basic types that have a length. But len is a special form
+  that is recognized by the compiler.
++ If a certain structure is never touched by len, the compiler might save the
+  length field from the underlying structure.
+``int, float, ord, chr``... are available as simple conversion functions.
+``int, float, str``... have a special meaning as a type inside of isinstance only.
++ methods and other class attributes do not change after startup
++ inheritance is supported
++ classes are first-class objects too
+  wrapped objects are borrowed from the object space. Just like in CPython,
+  code that needs e.g. a dictionary can use a wrapped dict and the object space
+  operations on it.
+This layout makes the number of types to take care about quite limited.
+Integer Types
+While implementing the integer type, we stumbled over the problem, that
+integers are quite in flux in CPython right now. Starting on Python 2.2,
+integers mutate into longs on overflow.  However, shifting to the left
+truncates up to 2.3 but extends to longs as well in 2.4.  By contrast, we need
+a way to perform wrap-around machine-sized arithmetic by default, while still
+being able to check for overflow when we need it explicitly.  Moreover, we need
+a consistent behavior before and after translation.
+We use normal integers for signed arithmetic.  It means that before
+translation we get longs in case of overflow, and after translation we get a
+silent wrap-around.  Whenever we need more control, we use the following
+  This special function should only be used with a single arithmetic operation
+  as its argument, e.g. ``z = ovfcheck(x+y)``.  Its intended meaning is to
+  perform the given operation in overflow-checking mode.
+  At run-time, in Python, the ovfcheck() function itself checks the result
+  and raises OverflowError if it is a ``long``.  But the code generators use
+  ovfcheck() as a hint: they replace the whole ``ovfcheck(x+y)`` expression
+  with a single overflow-checking addition in C.
+  ovfcheck_lshift(x, y) is a workaround for ovfcheck(x<<y), because the
+  latter doesn't quite work in Python prior to 2.4, where the expression
+  ``x<<y`` will never return a long if the input arguments are ints.  There is
+  a specific function ovfcheck_lshift() to use instead of some convoluted
+  expression like ``x*2**y`` so that code generators can still recognize it as
+  a single simple operation.
+  This function is used for wrap-around arithmetic.  It returns the lower bits
+  of its argument, masking away anything that doesn't fit in a C "signed long int".
+  Its purpose is, in Python, to convert from a Python ``long`` that resulted from a
+  previous operation back to a Python ``int``.  The code generators ignore
+  intmask() entirely, as they are doing wrap-around signed arithmetic all the time
+  by default anyway.  (We have no equivalent of the "int" versus "long int"
+  distinction of C at the moment and assume "long ints" everywhere.)
+  In a few cases (e.g. hash table manipulation), we need machine-sized unsigned
+  arithmetic.  For these cases there is the r_uint class, which is a pure
+  Python implementation of word-sized unsigned integers that silently wrap
+  around.  The purpose of this class (as opposed to helper functions as above)
+  is consistent typing: both Python and the annotator will propagate r_uint
+  instances in the program and interpret all the operations between them as
+  unsigned.  Instances of r_uint are special-cased by the code generators to
+  use the appropriate low-level type and operations.
+  Mixing of (signed) integers and r_uint in operations produces r_uint that
+  means unsigned results.  To convert back from r_uint to signed integers, use
+  intmask().
+Exception rules
+Exceptions are by default not generated for simple cases.::
+    #!/usr/bin/python
+        lst = [1,2,3,4,5]
+        item = lst[i]    # this code is not checked for out-of-bound access
+        try:
+            item = lst[i]
+        except IndexError:
+            # complain
+Code with no exception handlers does not raise exceptions (after it has been
+translated, that is.  When you run it on top of CPython, it may raise
+exceptions, of course). By supplying an exception handler, you ask for error
+checking. Without, you assure the system that the operation cannot fail.
+This rule does not apply to *function calls*: any called function is
+assumed to be allowed to raise any exception.
+For example::
+    x = 5.1
+    x = x + 1.2       # not checked for float overflow
+    try:
+        x = x + 1.2
+    except OverflowError:
+        # float result too big
+    z = some_function(x, y)    # can raise any exception
+    try:
+        z = some_other_function(x, y)
+    except IndexError:
+        # only catches explicitly-raised IndexErrors in some_other_function()
+        # other exceptions can be raised, too, and will not be caught here.
+The ovfcheck() function described above follows the same rule: in case of
+overflow, it explicitly raise OverflowError, which can be caught anywhere.
+Exceptions explicitly raised or re-raised will always be generated.
+PyPy is debuggable on top of CPython
+PyPy has the advantage that it is runnable on standard
+CPython.  That means, we can run all of PyPy with all exception
+handling enabled, so we might catch cases where we failed to
+adhere to our implicit assertions.
+.. _`wrapping rules`:
+Wrapping rules
+PyPy is made of Python source code at two levels: there is on the one hand
+*application-level code* that looks like normal Python code, and that
+implements some functionalities as one would expect from Python code (e.g. one
+can give a pure Python implementation of some built-in functions like
+``zip()``).  There is also *interpreter-level code* for the functionalities
+that must more directly manipulate interpreter data and objects (e.g. the main
+loop of the interpreter, and the various object spaces).
+Application-level code doesn't see object spaces explicitly: it runs using an
+object space to support the objects it manipulates, but this is implicit.
+There is no need for particular conventions for application-level code.  The
+sequel is only about interpreter-level code.  (Ideally, no application-level
+variable should be called ``space`` or ``w_xxx`` to avoid confusion.)
+Naming conventions
+* ``space``: the object space is only visible at
+  interpreter-level code, where it is by convention passed around by the name
+  ``space``.
+* ``w_xxx``: any object seen by application-level code is an
+  object explicitly managed by the object space.  From the
+  interpreter-level point of view, this is called a *wrapped*
+  object.  The ``w_`` prefix is used for any type of
+  application-level object.
+* ``xxx_w``: an interpreter-level container for wrapped
+  objects, for example a list or a dict containing wrapped
+  objects.  Not to be confused with a wrapped object that
+  would be a list or a dict: these are normal wrapped objects,
+  so they use the ``w_`` prefix.
+Operations on ``w_xxx``
+The core interpreter considers wrapped objects as black boxes.
+It is not allowed to inspect them directly.  The allowed
+operations are all implemented on the object space: they are
+called ``space.xxx()``, where ``xxx`` is a standard operation
+name (``add``, ``getattr``, ``call``, ``eq``...).  The list of
+standard operations is found in the large table near the end
+of ``pypy.interpreter.baseobjspace``.  These operations take
+wrapped arguments and return a wrapped result (or sometimes
+just None).
+Also note some helpers:
+* ``space.call_function(w_callable, ...)``: collects the given
+  (already-wrapped) arguments, builds a wrapped tuple for
+  them, and uses ``space.call()`` to perform the call.
+* ``space.call_method(w_object, 'method', ...)``: uses
+  ``space.getattr()`` to get the method object, and then
+  ``space.call_function()`` to invoke it.
+Building ``w_xxx`` objects
+From the core interpreter, wrapped objects are usually built as the result of
+an object space operation.  The ways to directly create a wrapped object are:
+* ``space.wrap(x)``: returns a wrapped object containing the
+  value ``x``.  Only works if ``x`` is either a simple value
+  (integer, float, string) or an instance of an internal
+  interpreter class (Function, Code, Frame...).
+* ``space.newlist([w_x, w_y, w_z...])``: returns a wrapped
+  list from a list of already-wrapped objects.
+* ``space.newtuple([w_x, w_y, w_z...])``: returns a wrapped
+  tuple from a list of already-wrapped objects.
+* ``space.newdict([])``: returns a new, empty wrapped
+  dictionary.  (The argument list can contain tuples ``(w_key,
+  w_value)`` but it seems that such a use is not common.)
+* ``space.newbool(x)``: returns ``space.w_False`` or
+  ``space.w_True`` depending on the truth value of ``x``.
+There are a few less common constructors, described in the
+comments at the end of ``pypy.interpreter.baseobjspace``.
+Constant ``w_xxx`` objects
+The object space holds a number of predefined wrapped objects.
+The most common ones are ``space.w_None`` and
+``space.w_XxxError`` for each exception class ``XxxError``
+(e.g. ``space.w_KeyError``, ``space.w_IndexError``, etc.).
+Inspecting and unwrapping ``w_xxx`` objects
+The most delicate operation is for the interpreter to inspect
+a wrapped object, which must be done via the object space.
+* ``space.is_true(w_x)``: checks if the given wrapped object
+  is considered to be ``True`` or ``False``.  You must never
+  use the truth-value of ``w_x`` directly; doing so (e.g.
+  writing ``if w_x:``) will give you an error reminding you of
+  the problem.
+* ``w_x == w_y`` or ``w_x is w_y``: DON'T DO THAT.  The only
+  half-official exception is to check if ``w_x`` contains a
+  wrapped ``None``: you can write ``w_x == space.w_None``.
+  Follow this rule; the case of ``None`` is easy to fix
+  globally later if we find out that we need to.  The
+  rationale for this rule is that there is no reason that two
+  wrappers are related in any way even if they contain what
+  looks like the same object at application-level.  To check
+  for equality, use ``space.is_true(space.eq(w_x, w_y))`` or
+  even better the short-cut ``space.eq_w(w_x, w_y)`` returning
+  directly a interpreter-level bool.  To check for identity,
+  use ``space.is_true(space.is_(w_x, w_y))`` or better
+  ``space.is_w(w_x, w_y)``.
+* ``space.unpackiterable(w_x)``: this helper iterates ``w_x``
+  (using ``space.iter()`` and ``space.next()``) and collects
+  the resulting wrapped objects in a list.  Of course, in
+  cases where iterating directly is better than collecting the
+  elements in a list first, you should use ``space.iter()``
+  and ``space.next()`` directly.
+* ``space.unwrap(w_x)``: inverse of ``space.wrap()``.
+  Attention!  Using ``space.unwrap()`` must be avoided
+  whenever possible, i.e. only use this when you are well
+  aware that you are cheating, in unit tests or bootstrapping
+  code.
+* ``space.interpclass_w(w_x)``: If w_x is a wrapped instance
+  of an interpreter class -- for example Function, Frame,
+  Cell, etc. -- return it unwrapped.  Otherwise return None.
+* ``space.int_w(w_x)``: If w_x is an application-level integer
+  or long which can be converted without overflow to an
+  integer, return an interpreter-level integer.  Otherwise
+  raise TypeError or OverflowError.
+* ``space.str_w(w_x)``: If w_x is an application-level string,
+  return an interpreter-level string.  Otherwise raise
+  TypeError.
+* ``space.float_w(w_x)``: If w_x is an application-level
+  float, integer or long, return interpreter-level float.
+  Otherwise raise TypeError or OverflowError in case of very
+  large longs.
+Remember that you can usually obtain the information you want
+by invoking operations or methods on the wrapped objects; e.g.
+``space.call_method(w_dict, 'iterkeys')`` returns a wrapped
+iterable that you can decode with ``space.unpackiterable()``.
+.. _`applevel-exceptions`: 
+Application-level exceptions
+Interpreter-level code can use exceptions freely.  However,
+all application-level exceptions are represented as an
+``OperationError`` at interpreter-level.  In other words, all
+exceptions that are potentially visible at application-level
+are internally an ``OperationError``.  This is the case of all
+errors reported by the object space operations
+(``space.add()`` etc.).
+To raise an application-level exception::
+    raise OperationError(space.w_XxxError, space.wrap("message"))
+To catch a specific application-level exception::
+    try:
+        ...
+    except OperationError, e:
+        if not e.match(space, space.w_XxxError):
+            raise
+        ...
+This construct catches all application-level exceptions, so we
+have to match it against the particular ``w_XxxError`` we are
+interested in and re-raise other exceptions.  The exception
+instance ``e`` holds two attributes that you can inspect:
+``e.w_type`` and ``e.w_value``.  Do not use ``e.w_type`` to
+match an exception, as this will miss exceptions that are
+instances of subclasses.
+We are thinking about replacing ``OperationError`` with a
+family of common exception classes (e.g. ``AppKeyError``,
+``AppIndexError``...) so that we can more easily catch them.
+The generic ``AppError`` would stand for all other
+application-level classes.
+.. _`modules`:
+Modules in PyPy
+Modules visible from application programs are imported from
+interpreter or application level files.  PyPy reuses almost all python
+modules of CPython's standard library, currently from version 2.3.4.  We
+sometimes need to `modify modules`_ and - more often - regression tests
+because they rely on implementation details of CPython.
+If we don't just modify an original CPython module but need to rewrite
+it from scratch we put it into `pypy/lib/`_ as a pure application level
+When we need access to interpreter-level objects we put the module into
+`pypy/module`_.  Such modules use a `mixed module mechanism`_
+which makes it convenient to use both interpreter- and applicationlevel
+parts for the implementation.  Note that there is no extra facility for
+pure-interpreter level modules because we haven't needed it so far.
+Determining the location of a module implementation
+You can interactively find out where a module comes from,
+here are examples for the possible locations::
+    >>>> import sys
+    >>>> sys.__file__
+    '/home/hpk/pypy-dist/pypy/module/sys/__init__.pyc'
+    >>>> import operator
+    >>>> operator.__file__
+    '/home/hpk/pypy-dist/pypy/lib/operator.py'
+    >>>> import types
+    t>>>> types.__file__
+    '/home/hpk/pypy-dist/lib-python/modified-2.3.4/types.py'
+    >>>> import os
+    faking <type 'posix.stat_result'>
+    faking <type 'posix.statvfs_result'>
+    >>>> os.__file__
+    '/home/hpk/pypy-dist/lib-python/2.3.4/os.py'
+    >>>>
+Module directories / Import order
+Here is the order in which PyPy looks up Python modules:
+    mixed interpreter/app-level builtin modules, such as
+    the ``sys`` and ``__builtin__`` module.
+*contents of PYTHONPATH*
+    lookup application level modules in each of the ``:`` separated
+    list of directories, specified in the ``PYTHONPATH`` environment
+    variable.
+    contains pure Python reimplementation of modules.
+    The files and tests that we have modified from the CPython library.
+    The unmodified CPython library. **Never ever checkin anything here**.
+.. _`modify modules`:
+Modifying a CPython library module or regression test
+Although PyPy is very compatible with CPython we sometimes need
+to change modules contained in our copy of the standard library,
+often due to the fact that PyPy works with all new-style classes
+by default and CPython has a number of places where it relies
+on some classes being old-style.
+If you want to change a module or test contained in ``lib-python/2.3.4``
+then make sure that you copy the file to our ``lib-python/modified-2.3.4``
+directory first.  In subversion commandline terms this reads::
+    svn cp lib-python/2.3.4/somemodule.py lib-python/modified-2.3.4/
+and subsequently you edit and commit
+``lib-python/modified-2.3.4/somemodule.py``.  This copying operation is
+important because it keeps the original CPython tree clean and makes it
+obvious what we had to change.
+.. _`mixed module mechanism`:
+.. _`mixed modules`:
+Implementing a mixed interpreter/application level Module
+If a module needs to access PyPy's interpreter level
+then it is implemented as a mixed module.
+Mixed modules are directories in `pypy/module`_ with an  `__init__.py`
+file containing specifications where each name in a module comes from.
+Only specified names will be exported to a Mixed Module's applevel
+application level definitions
+Application level specifications are found in the `appleveldefs`
+dictionary found in ``__init__.py`` files of directories in ``pypy/module``.
+For example, in `pypy/module/__builtin__/__init__.py`_ you find the following
+entry specifying where ``__builtin__.locals`` comes from::
+     ...
+     'locals'        : 'app_inspect.locals',
+     ...
+The ``app_`` prefix indicates that the submodule ``app_inspect`` is
+interpreted at application level and the wrapped function value for ``locals``
+will be extracted accordingly.
+interpreter level definitions
+Interpreter level specifications are found in the ``interpleveldefs``
+dictionary found in ``__init__.py`` files of directories in ``pypy/module``.
+For example, in `pypy/module/__builtin__/__init__.py`_ the following
+entry specifies where ``__builtin__.len`` comes from::
+     ...
+     'len'       : 'operation.len',
+     ...
+The ``operation`` submodule lives at interpreter level and ``len``
+is expected to be exposable to application level.  Here is
+the definition for ``operation.len()``::
+    def len(space, w_obj):
+        "len(object) -> integer\n\nReturn the number of items of a sequence or mapping."
+        return space.len(w_obj)
+Exposed interpreter level functions usually take a ``space`` argument
+and some wrapped values (see `wrapping rules`_) .
+You can also use a convenient shortcut in ``interpleveldefs`` dictionaries:
+namely an expression in parentheses to specify an interpreter level
+expression directly (instead of pulling it indirectly from a file)::
+    ...
+    'None'          : '(space.w_None)',
+    'False'         : '(space.w_False)',
+    ...
+The interpreter level expression has a ``space`` binding when
+it is executed.
+Testing modules in ``pypy/lib``
+You can go to the `pypy/lib/test2`_ directory and invoke the testing tool
+("py.test" or "python ../../test_all.py") to run tests against the
+pypy/lib hierarchy.  Note, that tests in `pypy/lib/test2`_ are allowed
+and encouraged to let their tests run at interpreter level although
+`pypy/lib/`_ modules eventually live at PyPy's application level.
+This allows us to quickly test our python-coded reimplementations
+against CPython.
+Testing modules in ``pypy/module``
+Simply change to ``pypy/module`` or to a subdirectory and `run the
+tests as usual`_.
+Testing modules in ``lib-python``
+In order to let CPython's regression tests run against PyPy
+you can switch to the `lib-python/`_ directory and run
+the testing tool in order to start compliance tests.
+(XXX check windows compatibility for producing test reports).
+Naming conventions and directory layout
+Directory and File Naming
+- directories/modules/namespaces are always **lowercase**
+- never use plural names in directory and file names
+- ``__init__.py`` is usually empty except for
+  ``pypy/objspace/*`` and ``pypy/module/*/__init__.py``.
+- don't use more than 4 directory nesting levels
+- keep filenames concise and completion-friendly.
+Naming of python objects
+- class names are **CamelCase**
+- functions/methods are lowercase and ``_`` separated
+- objectspace classes are spelled ``XyzObjSpace``. e.g.
+  - StdObjSpace
+  - FlowObjSpace
+- at interpreter level and in ObjSpace all boxed values
+  have a leading ``w_`` to indicate "wrapped values".  This
+  includes w_self.  Don't use ``w_`` in application level
+  python only code.
+Committing & Branching to the repository
+- write good log messages because several people
+  are reading the diffs.
+- if you add (text/py) files to the repository then please run
+  pypy/tool/fixeol in that directory.  This will make sure
+  that the property 'svn:eol-style' is set to native which
+  allows checkin/checkout in native line-ending format.
+- branching (aka "svn copy") of source code should usually
+  happen at ``svn/pypy/dist`` level in order to have a full
+  self-contained pypy checkout for each branch.   For branching
+  a ``try1`` branch you would for example do::
+    svn cp http://codespeak.net/svn/pypy/dist \
+           http://codespeak.net/svn/pypy/branch/try1
+  This allows to checkout the ``try1`` branch and receive a
+  self-contained working-copy for the branch.   Note that
+  branching/copying is a cheap operation with subversion, as it
+  takes constant time irrespective of the size of the tree.
+- To learn more about how to use subversion read `this document`_.
+.. _`this document`: svn-help.html
+.. _`using development tracker`:
+Using the development bug/feature tracker
+We have a `development tracker`_, based on Richard Jones'
+`roundup`_ application.  You can file bugs,
+feature requests or see what's going on
+for the next milestone, both from an E-Mail and from a
+web interface.
+use your codespeak login or register
+If you already committed to the PyPy source code, chances
+are that you can simply use your codespeak login that
+you use for subversion or for shell access.
+If you are not a commiter then you can still `register with
+the tracker`_ easily.
+modifying Issues from svn commit messages
+If you are committing something related to
+an issue in the development tracker you
+can correlate your login message to a tracker
+item by following these rules:
+- put the content of ``issueN STATUS`` on a single
+  new line
+- `N` must be an existing issue number from the `development tracker`_.
+- STATUS is one of::
+    unread
+    chatting
+    need-eg   (jargon for "need example")
+    in-progress
+    testing
+    done-cbb  (jargon for 'done, could-be-better')
+    resolved
+.. _`register with the tracker`: https://codespeak.net/issue/pypy-dev/user?@template=register
+.. _`development tracker`: http://codespeak.net/issue/pypy-dev/
+.. _`roundup`: http://roundup.sf.net
+.. _`testing in PyPy`:
+Testing in PyPy
+Our tests are based on the new `py.test`_ tool which lets you write
+unittests without boilerplate.  All tests of modules
+in a directory usually reside in a subdirectory **test**.  There are
+basically two types of unit tests:
+- **Interpreter Level tests**. They run at the same level as PyPy's
+  interpreter.
+- **Application Level tests**. They run at application level which means
+  that they look like straight python code but they are interpreted by PyPy.
+Both types of tests need an `objectspace`_ they can run with (the interpreter
+dispatches operations on objects to an objectspace).  If you run a test you
+can usually give the '-o' switch to select an object space.  E.g. '-o thunk'
+will select the thunk object space. The default is the `Standard Object Space`_
+which aims to implement unmodified Python semantics.
+.. _`standard object space`: objspace.html#standard-object-space
+.. _`objectspace`: architecture.html#objectspace
+.. _`py.test`: http://codespeak.net/py/current/doc/test.html
+Interpreter level tests
+You can write test functions and methods like this::
+    def test_something(space):
+        # use space ...
+    class TestSomething:
+        def test_some(self):
+            # use 'self.space' here
+Note that the prefix `test` for test functions and `Test` for test
+classes is mandatory.  In both cases you can import Python modules at
+module global level and use plain 'assert' statements thanks to the
+usage of the `py.test`_ tool.
+Application Level tests
+For testing the conformance and well-behavedness of PyPy it
+is often sufficient to write "normal" application-level
+Python code that doesn't need to be aware of any particular
+coding style or restrictions.  If we have a choice we often
+use application level tests which usually look like this::
+    def app_test_something():
+        # application level test code
+    class AppTestSomething:
+        def test_this(self):
+            # application level test code
+These application level test functions will run on top
+of PyPy, i.e. they have no access to interpreter details.
+You cannot use imported modules from global level because
+they are imported at interpreter-level while you test code
+runs at application level. If you need to use modules
+you have to import them within the test function.
+.. _`run the tests as usual`:
+Command line tool test_all
+You can run almost all of PyPy's tests by invoking::
+  python test_all.py
+which is a synonym for the general `py.test`_ utility
+located in the ``pypy`` directory.  For switches to
+modify test execution pass the ``-h`` option.
+Test conventions
+- adding features requires adding appropriate tests.  (It often even
+  makes sense to first write the tests so that you are sure that they
+  actually can fail.)
+- All over the pypy source code there are test/ directories
+  which contain unittests.  Such scripts can usually be executed
+  directly or are collectively run by pypy/test_all.py
+- each test directory needs a copy of pypy/tool/autopath.py which
+  upon import will make sure that sys.path contains the directory
+  where 'pypy' is in.
+.. _`change documentation and website`:
+Changing documentation and website
+documentation/website files in your local checkout
+Most of the PyPy's documentation and website is kept in
+`pypy/documentation` and `pypy/documentation/website` respectively.
+You can simply edit or add '.txt' files which contain ReST-markuped
+files.  Here is a `ReST quickstart`_ but you can also just look
+at the existing documentation and see how things work.
+.. _`ReST quickstart`: http://docutils.sourceforge.net/docs/rst/quickref.html
+Automatically test documentation/website changes
+.. _`docutils home page`:
+.. _`docutils`: http://docutils.sourceforge.net/
+We automatically check referential integrity and ReST-conformance.  In order to
+run the tests you need docutils_ installed.  Then go to the local checkout
+of the documentation directory and run the tests::
+    cd .../pypy/documentation
+    python ../test_all.py
+If you see no failures chances are high that your modifications at least
+don't produce ReST-errors or wrong local references.  A side effect of running
+the tests is that you have `.html` files in the documentation directory
+which you can point your browser to!
+Additionally, if you also want to check for remote references inside
+the documentation issue::
+    python ../test_all.py --checkremote
+which will check that remote URLs are reachable.
+.. include:: _ref.txt

File pypy/doc/confrest.py

View file
+from py.__.documentation.confrest import *
+class PyPyPage(Page): 
+    def fill(self):
+        super(PyPyPage, self).fill()
+        self.menubar[:] = html.div(
+            html.a("news", href="news.html", class_="menu"), " ",
+            html.a("doc", href="index.html", class_="menu"), " ",
+            html.a("contact", href="contact.html", class_="menu"), " ", 
+            html.a("getting-started", 
+                   href="getting-started.html", class_="menu"), " ",
+            html.a("issue", 
+                   href="https://codespeak.net/issue/pypy-dev/", 
+                   class_="menu"), 
+            " ", id="menubar")
+class Project(Project): 
+    title = "PyPy" 
+    stylesheet = 'style.css'
+    encoding = 'latin1' 
+    prefix_title = "PyPy"
+    logo = html.div(
+        html.a(
+            html.img(alt="PyPy", id="pyimg", 
+                     src="http://codespeak.net/pypy/img/py-web1.png", 
+                     height=110, width=149)))
+    Page = PyPyPage 

File pypy/doc/conftest.py

View file
+import py
+from py.__.documentation.conftest import Directory, DoctestText, ReSTChecker
+class PyPyDoctestText(DoctestText): 
+    def run(self): 
+        # XXX refine doctest support with respect to scoping 
+        return  
+    def execute(self, module, docstring): 
+        # XXX execute PyPy prompts as well 
+        l = []
+        for line in docstring.split('\n'): 
+            if line.find('>>>>') != -1: 
+                line = "" 
+            l.append(line) 
+        text = "\n".join(l) 
+        super(PyPyDoctestText, self).execute(module, text) 
+        #mod = py.std.types.ModuleType(self.fspath.basename, text) 
+        #self.mergescopes(mod, scopes) 
+        #failed, tot = py.std.doctest.testmod(mod, verbose=1)
+        #if failed:
+        #    py.test.fail("doctest %s: %s failed out of %s" %(
+        #                 self.fspath, failed, tot))
+class PyPyReSTChecker(ReSTChecker): 
+    DoctestText = PyPyDoctestText 
+class Directory(Directory): 
+    ReSTChecker = PyPyReSTChecker 

File pypy/doc/contact.txt

View file
+PyPy mailing lists
+* `development mailing list`_ for conceptual and coding discussions (low to medium traffic). 
+* `subversion commit mailing list`_ all updates to the trunk/branch source and documentation tree. 
+* `sprint mailing list`_ for people (interested in) attending upcoming sprints. 
+* `EU partner mailing list`_ for people involved with EU administrative details 
+.. _`sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint 
+.. _`subversion commit mailing list`: http://codespeak.net/mailman/listinfo/pypy-svn
+.. _`development mailing list`: http://codespeak.net/mailman/listinfo/pypy-dev
+.. _`EU partner mailing list`: http://codespeak.net/mailman/listinfo/pypy-funding 
+PyPy issue trackers 
+* `development bug/feature tracker`_ for filing bugs and feature requests. 
+* `EU milestone/issue tracking`_ for organizing work on EU-project milestones and deliverables 
+.. _`EU milestone/issue tracking`: https://codespeak.net/issue/pypy-eu/ 
+.. _`development bug/feature tracker`: https://codespeak.net/issue/pypy-dev/ 
+IRC Channel #pypy on freenode
+Many of the core developers are hanging out at #pypy on irc.freenode.net. 
+You are welcome to join and ask questions even more so if you are interested
+in participating in some parts of the PyPy project. You can find the logs of
+the channel here_.
+.. _here: http://nimrod.terra-link.net/pypy

File pypy/doc/contributor.txt

View file
+Contributors to PyPy
+Here is a list of developers who have committed to the PyPy source
+code base, ordered by number of commits (which is certainly not a very
+appropriate measure but it's something)::
+    Armin Rigo
+    Samuele Pedroni
+    Holger Krekel
+    Michael Hudson
+    Christian Tismer
+    Seo Sanghyeon
+    Alex Martelli
+    Stefan Schwarzer
+    Tomek Meka
+    Patrick Maupin
+    Carl Friedrich Bolz
+    Bob Ippolito
+    Anders Chrigstrom
+    Jacob Hallen
+    Marius Gedminas
+    Laura Creighton
+    Guido van Rossum
+    Richard Emslie
+    Ludovic Aubry
+    Adrien Di Mascio
+    Stephan Diehl
+    Dinu Gherman
+    Guenter Jantzen
+    Anders Lehmann
+    Rocco Moretti
+    Olivier Dormond
+    Brian Dorsey
+    Jonathan David Riehl
+    Andreas Friedge
+    Jens-Uwe Mager
+    Alan McIntyre
+    Lutz Paelike
+    Jacek Generowicz
+    Ben Young
+    Michael Chermside

File pypy/doc/ext-functions-draft.txt

View file
+example of interp-level code needing to somehow call an external function::
+    import os
+    def open(space, w_fname):
+        fname = space.str_w(fname)
+        fd = os.open(fname)
+        return space.wrap(fd)
+* os.open will be special-cased along all translation levels, there
+  will be a table mapping "sys calls" functions to the information
+  needed by the various levels.
+* annotation/builtin.py will take information from the common table 
+  to annotate the calls.
+* rtyper will replace the calls with for testing or dummy lowlevel
+  functions (ll_os_open for os.open), specified somehow trough the
+  table. The usual specialisation for low-level helpers will happem
+  but the original function will be attached as _callable to the
+  function pointer such that backends can recognize the original
+  function as from the table.
+* the backends will have implementations for these helper functions
+  (to allow writing such implementations we will need a way to assign
+  fixed names for some of the defined lltypes introduced by the rtyper)
+* XXX: integrate the following refinement:: 
+      interplevel: os.open()
+      annotation knows it returns an int. 
+      rtyper knows to subst by a dummy ll_os_open
+      def ll_os_open(...): 
+          fd = oslevel_open(...) 
+          if fd == -1: 
+            raise ... 

File pypy/doc/extradoc.txt

View file
+PyPy - talks and related projects 
+Talks and Presentations 
+* oscon2003-paper_ an early paper presented at Oscon 2003 describing 
+  what the PyPy project is about and why you should care. 
+* `Architecture introduction slides`_ a mostly up-to-date
+  introduction for the Amsterdam PyPy-Sprint Dec 2003. 
+* `EU funding for FOSS`_ talk on Chaos Communication
+  Conference in Berlin, Dec 2004. 
+* `PyCon 2005`_ animated slices, mostly reporting on the translator status.
+* `py lib slides`_ from the py lib talk at PyCon 2005 
+  (py is used as a support/testing library for PyPy). 
+.. _oscon2003-paper: http://codespeak.net/pypy/index.cgi?extradoc/talk/oscon2003-paper.html
+.. _`Architecture introduction slides`: http://codespeak.net/svn/pypy/extradoc/talk/amsterdam-sprint-intro.pdf
+.. _`EU funding for FOSS`: http://codespeak.net/svn/pypy/extradoc/talk/2004-21C3-pypy-EU-hpk.pdf
+.. _`py lib slides`: http://codespeak.net/svn/pypy/extradoc/talk/2005-pycon-py.pdf
+.. _`PyCon 2005`: http://codespeak.net/pypy/index.cgi?extradoc/talk/pypy-talk-pycon2005/README.html
+Related projects 
+* Dynamo_ showcased `transparent dynamic optimization`_
+  generating an optimized version of a binary program at runtime. 
+* Tailoring Dynamo_ to interpreter implementations and challenges -
+  Gregory Sullivan et. al., 
+  `Dynamic Native Optimization of Native Interpreters`_. IVME 03. 2003.
+* Stackless_ is a recursion-free version of Python.
+* Psyco_ is a just-in-time specializer for Python.
+* JikesRVM_ a research dynamic optimizing Java VM written in Java.
+* `Squeak`_ is a Smalltalk-80 implementation written in
+  Smalltalk, being used in `Croquet`_, an experimental 
+  distributed multi-user/multi-programmer virtual world. 
+* `LLVM`_ the low level virtual machine project. 
+* spyweb_ translates Python programs to Scheme. 
+* Jython_ is a Python implementation in Java.
+* `Iron Python`_ a new Python implementation compiling Python into 
+  Microsofts Common Language Runtime (CLR) Intermediate Language (IL).
+* `GNU lightning`_ generates assembly language at runtime. 
+.. _Stackless: http://stackless.com 
+.. _Psyco: http://psyco.sourceforge.net
+.. _Jython: http://www.jython.org
+.. _`Squeak`: http://www.squeak.org/
+.. _`Croquet`: http://www.opencroquet.org/
+.. _`Iron Python`: http://www.gotdotnet.com/workspaces/workspace.aspx?id=ad7acff7-ab1e-4bcb-99c0-57ac5a3a9742
+.. _`transparent dynamic optimization`: http://www.hpl.hp.com/techreports/1999/HPL-1999-77.pdf
+.. _Dynamo: http://www.hpl.hp.com/techreports/1999/HPL-1999-78.pdf
+.. _testdesign: http://codespeak.net/pypy/index.cgi?doc/testdesign.html
+.. _feasible: http://codespeak.net/pipermail/pypy-dev/2004q2/001289.html
+.. _rock: http://codespeak.net/pipermail/pypy-dev/2004q1/001255.html
+.. _spyweb: http://spyweb.hopto.org/
+.. _`GNU lightning`: http://www.gnu.org/software/lightning/lightning.html
+.. _LLVM: http://llvm.cs.uiuc.edu/
+.. _IronPython: http://www.python.org/pycon/dc2004/papers/9/
+.. _`Dynamic Native Optimization of Native Interpreters`: http://www.ai.mit.edu/~gregs/dynamorio.html
+.. _JikesRVM: http://jikesrvm.sf.net

File pypy/doc/faq.txt

View file
+Frequently Asked Questions
+How fast is PyPy?
+    As of June 2005, PyPy still runs on top of CPython and thus
+    is slower by a factor of 2000 compared to CPython.  We don't
+    know yet how fast it will be once we complete our translation
+    efforts, aimed at a first static self-contained low-level
+    translated version.

File pypy/doc/gc_planning.txt

View file
+   - write a simulator using arrays
+   - blocks are malloced with their size
+   - addresses are pointers memory locations:
+       * they are basically integers
+       * offsets are just regular integers
+       * their annotation is SomeAddress
+Address operations:
+construction / destruction:
+   - NULL: global object
+   - raw_malloc(size) --> address
+   - raw_free(addr)
+   - memcopy(addr1, addr2, size)
+memory access:
+   - reading: addr.signed[offset]
+   - writing: addr.signed[offset] = value
+   - datatypes:
+       * signed, unsigned, char, float(?), address
+open question / whish:
+it would be cool to do something like::
+    class A(object):
+        _raw_allocate_ = True
+        def method(self):
+            ...
+    a = A()
+    free_non_gc_object(a)
+    a.method() #-> crash
+it seems possible, will be messy
+Memory Layout:
+for now we go for the simple solution::
+                        +-<-<-<  object model sees only this
+                        V
+    +---------+---------+----------------------------+
+    | gc info | type id | object data                |
+    | int     | int     | whatever                   |
+    +---------+---------+----------------------------+

File pypy/doc/getting-started.txt

View file
+PyPy - Getting Started 
+.. contents::
+.. sectnum::
+.. _howtopypy: 
+Just the facts 
+getting & running the PyPy 0.6.1 release 
+    Note that the 0.6 release was broken and 
+    thus you find only the 0.6.1 release here. 
+Download one of the following release files and unpack it: 
+*pypy-0.6.1 (05/21/2005)*
+  * download one of `pypy-0.6.1.tar.bz2`_, `pypy-0.6.1.tar.gz`_ or
+    `pypy-0.6.1.zip`_ (windows line-endings) and unpack it 
+  * alternatively run ``svn co http://codespeak.net/svn/pypy/release/0.6.1 pypy-0.6.1`` 
+then change to the ``pypy-0.6.1`` directory
+and execute the following command line:: 
+    python pypy/bin/py.py 
+This will give you a PyPy prompt, i.e. a very compliant 
+Python interpreter implemented in Python.  Because this version
+of PyPy still runs on top of CPython, it runs around 2000
+times slower than the original CPython.  The release focus 
+really was on compliancy: PyPy passes around `90% of CPythons core 
+language regression tests`_. 
+Have a look at `interesting starting points`_ 
+for guidance on how to continue. 
+.. _`90% of CPythons core language regression tests`: http://codespeak.net/~hpk/pypy-testresult/ 
+.. _`pypy-0.6.1.tar.bz2`: http://codespeak.net/download/pypy/pypy-0.6.1.tar.bz2
+.. _`pypy-0.6.1.zip`: http://codespeak.net/download/pypy/pypy-0.6.1.zip
+.. _`pypy-0.6.1.tar.gz`: http://codespeak.net/download/pypy/pypy-0.6.1.tar.gz
+Svn-check out & run the latest PyPy as a two-liner
+If you want to play with the ongoing development PyPy version 
+you can check it out from the repository using subversion. Download 
+and install subversion_ if you don't allready have it. Then you can
+issue on the command line (DOS box or terminal)::
+    svn co http://codespeak.net/svn/pypy/dist pypy-dist 
+This will create a directory named ``pypy-dist``, and will get you the PyPy
+source in ``pypy-dist/pypy`` and documentation files in
+After checkout you can get a PyPy interpreter via:: 
+    python pypy-dist/pypy/bin/py.py
+have fun :-) 
+We have some `help on installing subversion`_ for PyPy. 
+Have a look at `interesting starting points`_ 
+for some guidance on how to continue. 
+.. _`help on installing subversion`: svn-help.html
+.. _subversion: svn-help.html
+Understanding PyPy's architecture
+For in-depth information about architecture and coding documentation 
+head over to the `documentation section`_ where you'll find lots of 
+interesting information.  Additionally, in true hacker spirit, you 
+may just `start reading sources`_ . 
+.. _`documentation section`: index.html 
+running all of PyPy's tests  
+If you want to see if PyPy works on your machine/platform 
+you can simply run PyPy's large test suite with:: 
+    python pypy/test_all.py 
+test_all.py is just another name for `py.test`_ which is the
+testing tool that we are using and enhancing for PyPy.   
+Filing bugs or feature requests 
+You may file `bug reports`_ on our issue tracker which is
+also accessible through the 'issues' top menu of 
+the PyPy website.  `using the development tracker`_ has 
+more detailed information on specific features of the tracker. 
+.. _`using the development tracker`: coding-guide.html#using-development-tracker
+.. _`interesting starting points`: 
+Interesting Starting Points in PyPy 
+The following assumes that you have successfully downloaded and extracted the
+PyPy release or have checked out PyPy using svn. It assumes that you are in
+the top level directory of the PyPy source tree, e.g. pypy-x.x (if you
+got a release) or pypy-dist (if you checked out the most recent version using
+Main entry point  
+The py.py interpreter
+To start interpreting Python with PyPy, use Python 2.3 or greater::
+    cd pypy/bin
+    python py.py
+After a few seconds (remember: this is running on top of CPython), 
+you should be at the PyPy prompt, which is the same as the Python 
+prompt, but with an extra ">".
+Now you are ready to start running Python code.  Most Python
+modules should work if they don't involve CPython extension 
+modules.  Here is an example of determining PyPy's performance 
+in pystones:: 
+    >>>> from test import pystone 
+    >>>> pystone.main(10)
+Note that this is a slightly modified version of pystone -- the
+original version does not accept the parameter to main().  The
+parameter is the number of loops to run through the test, and the
+default is 50000, which is far too many to run in a reasonable time
+on the current PyPy implementation.
+py.py options
+To list the PyPy interpreter command line options, type::
+    cd pypy/bin
+    python py.py --help
+As an example of using PyPy from the command line, you could type::
+    python py.py -c "from test import pystone; pystone.main(10)"
+Alternatively, as with regular Python, you can simply give a
+script name on the command line::
+    python py.py ../../lib-python/modified-2.3.4/test/pystone.py 10
+special PyPy features 
+Interpreter-level console
+There are a few extra features of the PyPy console: If you press
+<Ctrl-C> on the console you enter the interpreter-level console, a
+usual CPython console.  You can then access internal objects of PyPy
+(e.g. the object space) and any variables you have created on the PyPy
+prompt with the prefix ``w_``:: 
+    >>>> a = 123
+    >>>> <Ctrl-C>
+    *** Entering interpreter-level console ***
+    >>> w_a
+    W_IntObject(123)
+Note that the prompt of the interpreter-level console is only '>>>' since
+it runs on CPython level. If you want to return to PyPy, press <Ctrl-D> (under
+Linux) or <Ctrl-Z>, <Enter> (under Windows).
+You may be interested in reading more about the distinction between
+`interpreter-level and app-level`_.
+.. _`interpreter-level and app-level`: architecture.html#interpreter-level
+.. _`trace example`: 
+Tracing bytecode and operations on objects
+You can use the trace object space to monitor the interpretation
+of bytecodes in connection with object space operations.  To enable 
+it, set ``__pytrace__=1`` on the interactive PyPy console:: 
+    >>>> __pytrace__ = 1
+    Tracing enabled
+    >>>> a = 1 + 2
+    |- <<<< enter <inline>a = 1 + 2 @ 1 >>>>
+    |- 0    LOAD_CONST    0 (W_IntObject(1))
+    |- 3    LOAD_CONST    1 (W_IntObject(2))
+    |- 6    BINARY_ADD
+      |-    add(W_IntObject(1), W_IntObject(2))   -> W_IntObject(3)
+    |- 7    STORE_NAME    0 (a)
+      |-    setitem(W_DictObject([<Entry 3098577608L,W_StringObject('__builtins__'),
+                    <pypy.module..., W_StringObject('a'), W_IntObject(3))   
+            -> <W_NoneObject()>
+    |-10    LOAD_CONST    2 (<W_NoneObject()>)
+    |-13    RETURN_VALUE
+    |- <<<< leave <inline>a = 1 + 2 @ 1 >>>>
+lazily computed objects 
+One of the original features provided by PyPy is the "thunk" 
+object space, providing lazily-computed objects in a fully
+transparent manner::
+    cd pypy/bin
+    python py.py -o thunk
+    >>>> def longcomputation(lst):
+    ....     print "computing..."
+    ....     return sum(lst)
+    .... 
+    >>>> x = thunk(longcomputation, range(5))
+    >>>> y = thunk(longcomputation, range(10))
+from the application perspective, ``x`` and ``y`` represent 
+exactly the objects being returned by the ``longcomputation()``
+invocations.  You can put these objects into a dictionary 
+without triggering the computation:: 
+    >>>> d = {5: x, 10: y}