Clone wiki

compatibility / c-api


PyPy has some support for extensions based on Python's C-API in the form of CPyExt. See the list of C extensions known to work.

From PyPy's Python compatibility page:

PyPy has alpha-level support for the CPython C API, however, as of 1.4.1 release this feature is not yet complete. Most libraries will require a bit of effort to work, but there are known success stories. Check out PyPy blog for updates.

C extensions need to be recompiled for PyPy in order to work. Depending on your build system, it might work out of the box or will be slightly harder.

Relevant blog posts:

Given that PyPy can't inline or JIT compile the code, it has considerable overhead in calling C-API extensions. See the Shed Skin page for an example of the relative slowness of C-API-based code in PyPy.

There is a TODO for CPyExt.

PyPy's EuroPython 2010 talk covers CPyExt, here's the raw ReST of the slides:



- CPython extension modules in PyPy

- ``pypy-c build``

- included in PyPy 1.3

- still beta

- 50% of the CPython API is supported

  * enough for 90% of extension modules


- C API written in Python!

- Testable on top of an interpreted

- Written on top of the object space

- Source compatibility

  * PyString_AS_STRING is actually a function call (instead of a macro)


.. sourcecode:: python

    @cpython_api([PyObject], Py_ssize_t, error=-1)
    def PyDict_Size(space, w_obj):
        return space.int_w(space.len(w_obj))



- It was not supposed to work!

  * different garbage collector

  * no "borrowed reference"

  * all the PyTypeObject slots

- *not* faster than python code!

- PyObject contains ob_type and ob_refcnt

  * The "abstract object interface" is used.

- Some objects contain more:

  * PyString_AsString() must keep the buffer alive at a fixed location

  * PyTypeObject exposes all its fields

The Reference Counting Issue

- pypy uses a moving garbage collector, starts with static roots to
  find objects.

- CPython objects don't move, and PyObject* can point to deallocated

- cpyext builds PyObject as proxies to the "real" interpreter objects

- one dictionary lookup each time the boundary is crossed

- More tricks needed for borrowing references

  * The object lifetime is tied to its container.

  * "out of nothing" borrowed references are kept until the end of the
    current pypy->C call.

supported modules

- Known to work (after small patches):

  * wxPython

  * _sre

  * PyCrypto

  * PIL

  * cx_Oracle

  * MySQLdb

  * sqlite

Why your module will crash



.. sourcecode:: c

     static PyObject *myException;

     void init_foo() {
         myException = PyException_New(...);
         Py_AddModule(m, myException); // steals a reference

         PyErr_SetString(myException, "message"); // crash


wxPython on PyPy (1)

.. image:: wxpython1.png
   :scale: 30

wxPython on PyPy (2)

.. image:: wxpython2.png
   :scale: 30

performance (1)


.. sourcecode:: python

    from cx_Oracle import connect, STRING
    c = connect('scott/tiger@db')
    cur = c.cursor()
    var = cur.var(STRING)

    def f():
        for i in range(10000):
            var.setvalue(0, str(i))

.. sourcecode:: bash

    python -m timeit -s "from test_oracle import f" "f()"
    python2.6:    8.25 msec per loop
    pypy-c:     161    msec per loop
    pypy-c-jit: 121    msec per loop


performance (2)

Compare with:


.. sourcecode:: python

    def f():
        for i in range(10000):
            x = str(i)
            y = int(x)

.. sourcecode:: bash

    python2.6:    8.18 msec per loop
    pypy-c-jit:   1.22 msec per loop


Future developments

- Some care about speed

  * The JIT can help to remove (some of) the overhead

- Fill missing API functions (when needed)

- Better support of the PyTypeObject slots

- Think about threads and the GIL

- Think about reference cycles