M2Crypto fails to build, missing C-API support

Issue #2267 new
Luis Sebastian Urrutia Fuentes
created an issue

I clonned the last commit from m2crypto repo and run the installation with pypy and virtualenv.

m2crypto: https://gitlab.com/m2crypto/m2crypto

This is the log

(venv) root@back-front-node-dev:~/m2crypto# pypy setup.py install
running install
Checking .pth file support in /home/luisurrutiaf/venv/site-packages/
/home/luisurrutiaf/venv/bin/pypy -E -c pass
TEST PASSED: /home/luisurrutiaf/venv/site-packages/ appears to support .pth files
running bdist_egg
running egg_info
writing M2Crypto.egg-info/PKG-INFO
writing dependency_links to M2Crypto.egg-info/dependency_links.txt
writing top-level names to M2Crypto.egg-info/top_level.txt
reading manifest file 'M2Crypto.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.dat' under directory 'tests'
warning: no files found matching '*.p7*' under directory 'tests'
warning: no previously-included files found matching 'M2Crypto/_m2crypto.py'
writing manifest file 'M2Crypto.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
building 'M2Crypto.__m2crypto' extension
swigging SWIG/_m2crypto.i to SWIG/_m2crypto_wrap.c
swig -python -D__x86_64__ -I/home/luisurrutiaf/venv/include -I/usr/include -I/usr/include/openssl -includeall -modern -builtin -outdir build/lib.linux-x86_64-2.7/M2Crypto -o SWIG/_m2crypto_wrap.c SWIG/_m2
crypto.i
SWIG/_bio.i:64: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_rand.i:21: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_evp.i:169: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_dh.i:36: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_rsa.i:43: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_dsa.i:31: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_ssl.i:241: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_ssl.i:242: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_x509.i:323: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_pkcs7.i:44: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_pkcs7.i:44: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_util.i:11: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_ec.i:111: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_engine.i:168: Warning 454: Setting a pointer/reference variable may leak memory.
cc -O2 -fPIC -Wimplicit -I/home/luisurrutiaf/venv/include -I/usr/include -I/usr/include/openssl -I/home/luisurrutiaf/m2crypto/SWIG -c SWIG/_m2crypto_wrap.c -o build/temp.linux-x86_64-2.7/SWIG/_m2crypto_wrap.o -DTHREADING
SWIG/_m2crypto_wrap.c:3203:25: error: unknown type name ‘PyDescrObject’
 SwigPyStaticVar_dealloc(PyDescrObject *descr) {
                         ^
SWIG/_m2crypto_wrap.c:3211:22: error: unknown type name ‘PyGetSetDescrObject’
 SwigPyStaticVar_repr(PyGetSetDescrObject *descr) {
                      ^
SWIG/_m2crypto_wrap.c: In function ‘SwigPyStaticVar_traverse’:
SWIG/_m2crypto_wrap.c:3222:3: error: unknown type name ‘PyDescrObject’
   PyDescrObject *descr;
   ^
SWIG/_m2crypto_wrap.c:3223:12: error: ‘PyDescrObject’ undeclared (first use in this function)
   descr = (PyDescrObject *)self;
            ^
SWIG/_m2crypto_wrap.c:3223:12: note: each undeclared identifier is reported only once for each function it appears in
SWIG/_m2crypto_wrap.c:3223:27: error: expected expression before ‘)’ token
   descr = (PyDescrObject *)self;
                           ^
In file included from /home/luisurrutiaf/venv/include/Python.h:86:0,
                 from SWIG/_m2crypto_wrap.c:132:
SWIG/_m2crypto_wrap.c:940:43: error: expected expression before ‘)’ token
 #define PyDescr_TYPE(x) (((PyDescrObject *)(x))->d_type)
                                           ^
/home/luisurrutiaf/venv/include/object.h:555:21: note: in definition of macro ‘Py_VISIT’
                 if (op) {                                               \
                     ^
SWIG/_m2crypto_wrap.c:3224:24: note: in expansion of macro ‘PyDescr_TYPE’
   Py_VISIT((PyObject*) PyDescr_TYPE(descr));
                        ^
/home/luisurrutiaf/venv/include/object.h:557:29: error: ‘vret’ undeclared (first use in this function)
                         if (vret)                                       \
                             ^
SWIG/_m2crypto_wrap.c:3224:3: note: in expansion of macro ‘Py_VISIT’
   Py_VISIT((PyObject*) PyDescr_TYPE(descr));
   ^

SWIG/_m2crypto_wrap.c: At top level:
SWIG/_m2crypto_wrap.c:3229:21: error: unknown type name ‘PyGetSetDescrObject’
 SwigPyStaticVar_get(PyGetSetDescrObject *descr, PyObject *obj, PyObject *SWIGUNUSEDPARM(type)) {
                     ^
SWIG/_m2crypto_wrap.c:3241:21: error: unknown type name ‘PyGetSetDescrObject’
 SwigPyStaticVar_set(PyGetSetDescrObject *descr, PyObject *obj, PyObject *value) {
                     ^
SWIG/_m2crypto_wrap.c: In function ‘SwigPyStaticVar_Type’:
SWIG/_m2crypto_wrap.c:3292:14: error: ‘PyGetSetDescrObject’ undeclared (first use in this function)
       sizeof(PyGetSetDescrObject),
              ^
SWIG/_m2crypto_wrap.c:3294:19: error: ‘SwigPyStaticVar_dealloc’ undeclared (first use in this function)
       (destructor)SwigPyStaticVar_dealloc,      /* tp_dealloc */
                   ^
SWIG/_m2crypto_wrap.c:3299:17: error: ‘SwigPyStaticVar_repr’ undeclared (first use in this function)
       (reprfunc)SwigPyStaticVar_repr,           /* tp_repr */
                 ^
SWIG/_m2crypto_wrap.c:3322:21: error: ‘SwigPyStaticVar_get’ undeclared (first use in this function)
       (descrgetfunc)SwigPyStaticVar_get,        /* tp_descr_get */
                     ^
SWIG/_m2crypto_wrap.c:3323:21: error: ‘SwigPyStaticVar_set’ undeclared (first use in this function)
       (descrsetfunc)SwigPyStaticVar_set,        /* tp_descr_set */
                     ^
SWIG/_m2crypto_wrap.c: At top level:
SWIG/_m2crypto_wrap.c:3357:1: error: unknown type name ‘PyGetSetDescrObject’
 SWIGINTERN PyGetSetDescrObject *
 ^
SWIG/_m2crypto_wrap.c: In function ‘SwigPyStaticVar_new_getset’:
SWIG/_m2crypto_wrap.c:3360:3: error: unknown type name ‘PyGetSetDescrObject’
   PyGetSetDescrObject *descr;
   ^

SWIG/_m2crypto_wrap.c:3361:12: error: ‘PyGetSetDescrObject’ undeclared (first use in this function)
   descr = (PyGetSetDescrObject *)PyType_GenericAlloc(SwigPyStaticVar_Type(), 0);
            ^
SWIG/_m2crypto_wrap.c:3361:33: error: expected expression before ‘)’ token
   descr = (PyGetSetDescrObject *)PyType_GenericAlloc(SwigPyStaticVar_Type(), 0);
                                 ^
SWIG/_m2crypto_wrap.c:940:28: error: ‘PyDescrObject’ undeclared (first use in this function)
 #define PyDescr_TYPE(x) (((PyDescrObject *)(x))->d_type)
                            ^
SWIG/_m2crypto_wrap.c:3364:3: note: in expansion of macro ‘PyDescr_TYPE’
   PyDescr_TYPE(descr) = type;
   ^
SWIG/_m2crypto_wrap.c:940:43: error: expected expression before ‘)’ token
 #define PyDescr_TYPE(x) (((PyDescrObject *)(x))->d_type)
                                           ^
SWIG/_m2crypto_wrap.c:3364:3: note: in expansion of macro ‘PyDescr_TYPE’
   PyDescr_TYPE(descr) = type;
   ^
SWIG/_m2crypto_wrap.c:941:43: error: expected expression before ‘)’ token
 #define PyDescr_NAME(x) (((PyDescrObject *)(x))->d_name)
                                           ^
SWIG/_m2crypto_wrap.c:3365:3: note: in expansion of macro ‘PyDescr_NAME’
   PyDescr_NAME(descr) = PyString_InternFromString(getset->name);
   ^
SWIG/_m2crypto_wrap.c:3366:8: error: request for member ‘d_getset’ in something not a structure or union
   descr->d_getset = getset;
        ^

SWIG/_m2crypto_wrap.c:941:43: error: expected expression before ‘)’ token
 #define PyDescr_NAME(x) (((PyDescrObject *)(x))->d_name)
                                           ^
SWIG/_m2crypto_wrap.c:3367:7: note: in expansion of macro ‘PyDescr_NAME’
   if (PyDescr_NAME(descr) == NULL) {
       ^
SWIG/_m2crypto_wrap.c: In function ‘x509v3_lhash’:
SWIG/_m2crypto_wrap.c:7123:5: warning: return from incompatible pointer type [enabled by default]
     return lh_new(NULL, NULL); /* Should probably be lh_CONF_VALUE_new but won't compile. */
     ^
SWIG/_m2crypto_wrap.c: In function ‘_wrap_delete__STACK_closure’:
SWIG/_m2crypto_wrap.c:28071:1: warning: implicit declaration of function ‘PyType_IS_GC’ [-Wimplicit-function-declaration]
 SWIGPY_DESTRUCTOR_CLOSURE(_wrap_delete__STACK)
 ^
SWIG/_m2crypto_wrap.c: In function ‘init__m2crypto’:
SWIG/_m2crypto_wrap.c:29722:3: error: unknown type name ‘PyGetSetDescrObject’
   PyGetSetDescrObject *static_getset;
   ^
SWIG/_m2crypto_wrap.c:29776:3: warning: implicit declaration of function ‘PyDescr_NewGetSet’ [-Wimplicit-function-declaration]
   this_descr = PyDescr_NewGetSet(SwigPyObject_type(), &this_getset_def);
   ^
SWIG/_m2crypto_wrap.c:29776:14: warning: assignment makes pointer from integer without a cast [enabled by default]
   this_descr = PyDescr_NewGetSet(SwigPyObject_type(), &this_getset_def);
              ^
SWIG/_m2crypto_wrap.c:29780:17: warning: assignment makes pointer from integer without a cast [enabled by default]
   thisown_descr = PyDescr_NewGetSet(SwigPyObject_type(), &thisown_getset_def);
                 ^
error: command 'cc' failed with exit status 1

Pypy is the one from squeaky/portable-pypy (https://bitbucket.org/squeaky/portable-pypy) with version 5.0.1. - pypy-5.0.1-linux_x86_64-portable.tar.bz2

Comments (14)

  1. mattip

    PyGetSetDescrObject is not implemented on this release. This particular C-API call will be supported when we merge the cpyext-ext branch. Note that the C-API interface to PyPy is slower than using cffi. You might want to check out the pycrypto package instead, which uses cffi to provide many of the same services

  2. Matěj Cepl

    You might want to check out the pycrypto package instead, which uses cffi to provide many of the same services.

    Just to note that the huge difference between pycrypto and M2Crypto (of which I am a maintainer, so yes, I am biased) is that the latter uses OpenSSL (well-worn, battled, and tested in the real life) for all security algorithms, whereas pycrypto does it by its own.

    Of course, it doesn't help when PyPy doesn't support C-API, because without that M2Crypto is completely useless, and there is no way (short of complete rewrite) to make it work through cffi interface.

  3. mattip

    While I am not sure that swig-generated code will play nicely with PyPy, we actually are quite close to being able to test this package.

    On the latest nightly we are down to these missing functions for the build to successfully run to completion

    _PyObject_GC_UNTRACK - seems like a cpython-specific call (like a weak reference?) I guess we could just make it a NOOP?

    PyType_IS_GC - should be easy enough to copy the macro, fwiw.

    PyDescr_NewGetSet - we have a non-exported function by this name on the cpyext-ext branch, it needs a test

  4. Codescu Radu

    I have successfully install M2Crypto on pypy 5.8.0. But the problem is when I try to import M2Crypto, I get the following error:

    >>>> import M2Crypto
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/radu/Tasks/issues_2267/venv/site-packages/M2Crypto/__init__.py", line 26, in <module>
        from M2Crypto import (ASN1, AuthCookie, BIO, BN, DH, DSA, EVP, Engine, Err,
      File "/home/radu/Tasks/issues_2267/venv/site-packages/M2Crypto/ASN1.py", line 15, in <module>
        from M2Crypto import BIO, m2, util
      File "/home/radu/Tasks/issues_2267/venv/site-packages/M2Crypto/BIO.py", line 10, in <module>
        from M2Crypto import m2, util
      File "/home/radu/Tasks/issues_2267/venv/site-packages/M2Crypto/m2.py", line 30, in <module>
        from M2Crypto.m2crypto import *
      File "/home/radu/Tasks/issues_2267/venv/site-packages/M2Crypto/m2crypto.py", line 26, in <module>
        _m2crypto = swig_import_helper()
      File "/home/radu/Tasks/issues_2267/venv/site-packages/M2Crypto/m2crypto.py", line 22, in swig_import_helper
        _mod = imp.load_module('_m2crypto', fp, pathname, description)
    ImportError: unable to load extension module '/home/radu/Tasks/issues_2267/venv/site-packages/M2Crypto/_m2crypto.pypy-41.so':
     /home/radu/Tasks/issues_2267/venv/site-packages/M2Crypto/_m2crypto.pypy-41.so: undefined symbol: PyDescr_NewGetSet
    
  5. mattip

    It turns out we do not properly expose PyDescr_NewGetSet, in spite of what I wrote above. While the change to expose the function is simple, I fear the next step, assigning the new PyGetSetDescrObject into the object's tp_dict, will not actually update anything in the python app-level object since PyType_Ready has already been called.

    In short, we need an appropriate test for common usage of PyDescr_NewGetSet before we can fix this.

  6. mattip

    @Matěj Cepl much of the C Python API is supported by PyPy, but a corner used by SWIG unfortunately is not yet supported, especially pieces that touch the exposed C structures after calling PyType_Ready. PyDescr_NewGetSet is particularily difficult to support, it modifies the class's __dict__ via direct manipulation of the tp_dict C structure which is very difficult to detect in PyPy.

  7. Codescu Radu

    Hello,

    I have tried to expose the function PyDescr_NewGetSet as it is defined in pypy/module/cpyext/typeobject.py.

    After I expose the function, you can see changes in the diff file attached, I get an error at translation. I attached the file with the error I got.

    Also there is a small error in the function in typeobject.py, parameters are swapped, as you can see in docs, first parameter should be type and not getset.

    Is there another way to do it? I have no idea what is the cause for the error.

  8. Log in to comment