Support pybind11 in conjunction with PyPy's cpyext: issue with heap types

Issue #2434 resolved
Wenzel Jakob
created an issue

Hi,

I'm the maintainer of pybind11, which is a modern library for creating bindings to C++ code (similar in spirit to Boost.Python). Hearing of the cpyext improvements in the latest release, I tried to see if pybind11 would work with the supported CPython API in PyPy. This would be huge, since it immediately make many projects available in PyPy.

The good news: I just had to make a few minor changes to get a few simple examples to compile. This is all in the WIP branch in pybind11 here: https://github.com/pybind/pybind11/pull/521

The bad news: I'm running into fundamental issues involving creation of types.

To reproduce:

$ git clone https://github.com/wjakob/pybind11
$ cd pybind11
$ cat > test.cpp <<EOF
#include <pybind11/pybind11.h>

namespace py = pybind11;

PYBIND11_PLUGIN(test) {
    py::module m("test");

    struct Test { };
    py::class_<Test>(m, "Test");

    return m.ptr();
}
EOF
$ g++ test.cpp -g3 -std=c++11 -I <path-to-pypy-include> -I include -undefined dynamic_lookup -shared -rdynamic -o test.so

Loading the module from PyPy leads to the following error:

$ pypy
Python 2.7.12 (aff251e543859ce4508159dd9f1a82a2f553de00, Nov 13 2016, 01:57:41)
[PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>> import cpyext
>>>> pyext.load_module('test.so', 'test')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: global name 'pyext' is not defined
>>>> cpyext.load_module('test.so', 'test')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: TypeError: can't set attributes on type object 'Test'

The error message is thrown when pybind11 basically calls (via CPython API)

setattr(Test, '__module__', 'test')

Digging into the PyPy source code reveals the following source of this error message:

def setdictvalue(self, space, name, w_value):
    if not self.is_heaptype():
        raise oefmt(space.w_TypeError,
                    "can't set attributes on type object '%N'", self)

So PyPy believes that Test is not a heap type. Which is confusing, because cpyext believes it is.

The CPython API calls boil down to:

PyHeapTypeObject *type = ((PyHeapTypeObject*) PyType_Type.tp_alloc(&PyType_Type, 0)
type->ht_type.tp_flags |= Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HEAPTYPE;

.. fill out remaining type fields like name, docs, ...

PyType_Ready(&type->ht_type);
PyObject_SetAttrString(type->ht_type, "__module__", ...); // BOOM, this fails

Interestingly, the type instance returned by tp_alloc already has the heap type flag set, and it is still set after the PyType_Ready call. I can only assume that PyPy has its own heap type flag that somehow isn't synchronized with the types created by cpyext.

Comments (5)

  1. Wenzel Jakob reporter

    Great, thank you -- I was able to compile PyPy myself and can confirm that this addressed the issue. I'll likely have a few more followup bug reports for the remaining issues on the way towards full PyPy/pybind11 integration.

  2. Log in to comment