c-api

C-API

PyPy has some support for extensions based on Python's C-API in the form of CPyExt. See the list of C extensions known to work.

From PyPy's Python compatibility page:


PyPy has alpha-level support for the CPython C API, however, as of 1.4.1 release this feature is not yet complete. Most libraries will require a bit of effort to work, but there are known success stories. Check out PyPy blog for updates.

C extensions need to be recompiled for PyPy in order to work. Depending on your build system, it might work out of the box or will be slightly harder.


Relevant blog posts:

Given that PyPy can't inline or JIT compile the code, it has considerable overhead in calling C-API extensions. See the Shed Skin page for an example of the relative slowness of C-API-based code in PyPy.

There is a TODO for CPyExt.

PyPy's EuroPython 2010 talk covers CPyExt, here's the raw ReST of the slides:

cpyext

cpyext
------

- CPython extension modules in PyPy

- ``pypy-c setup.py build``

- included in PyPy 1.3

- still beta

- 50% of the CPython API is supported

  * enough for 90% of extension modules

features
--------

- C API written in Python!

- Testable on top of an interpreted py.py

- Written on top of the object space

- Source compatibility

  * PyString_AS_STRING is actually a function call (instead of a macro)

|small|

.. sourcecode:: python

    @cpython_api([PyObject], Py_ssize_t, error=-1)
    def PyDict_Size(space, w_obj):
        return space.int_w(space.len(w_obj))

|end_small|

implementation
--------------

- It was not supposed to work!

  * different garbage collector

  * no "borrowed reference"

  * all the PyTypeObject slots

- *not* faster than python code!

- PyObject contains ob_type and ob_refcnt

  * The "abstract object interface" is used.

- Some objects contain more:

  * PyString_AsString() must keep the buffer alive at a fixed location

  * PyTypeObject exposes all its fields


The Reference Counting Issue
----------------------------

- pypy uses a moving garbage collector, starts with static roots to
  find objects.

- CPython objects don't move, and PyObject* can point to deallocated
  memory.

- cpyext builds PyObject as proxies to the "real" interpreter objects

- one dictionary lookup each time the boundary is crossed

- More tricks needed for borrowing references

  * The object lifetime is tied to its container.

  * "out of nothing" borrowed references are kept until the end of the
    current pypy->C call.




supported modules
------------------

- Known to work (after small patches):

  * wxPython

  * _sre

  * PyCrypto

  * PIL

  * cx_Oracle

  * MySQLdb

  * sqlite


Why your module will crash
---------------------------

Likely:

|small|

.. sourcecode:: c

     static PyObject *myException;

     void init_foo() {
         myException = PyException_New(...);
         Py_AddModule(m, myException); // steals a reference
     }

     {
         PyErr_SetString(myException, "message"); // crash
     }

|end_small|

wxPython on PyPy (1)
---------------------

.. image:: wxpython1.png
   :scale: 30

wxPython on PyPy (2)
---------------------

.. image:: wxpython2.png
   :scale: 30

performance (1)
---------------

|small|

.. sourcecode:: python

    from cx_Oracle import connect, STRING
    c = connect('scott/tiger@db')
    cur = c.cursor()
    var = cur.var(STRING)

    def f():
        for i in range(10000):
            var.setvalue(0, str(i))
            var.getvalue(0)


.. sourcecode:: bash

    python -m timeit -s "from test_oracle import f" "f()"
    python2.6:    8.25 msec per loop
    pypy-c:     161    msec per loop
    pypy-c-jit: 121    msec per loop

|end_small|

performance (2)
---------------


Compare with:

|small|

.. sourcecode:: python

    def f():
        for i in range(10000):
            x = str(i)
            y = int(x)

.. sourcecode:: bash

    python2.6:    8.18 msec per loop
    pypy-c-jit:   1.22 msec per loop

|end_small|

Future developments
-------------------

- Some care about speed

  * The JIT can help to remove (some of) the overhead

- Fill missing API functions (when needed)

- Better support of the PyTypeObject slots

- Think about threads and the GIL

- Think about reference cycles

Updated

Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.