ctypes backend: `ffi.gc`'d pointers do not implicitly convert to their original type

Issue #282 resolved
dgelessus
created an issue

This is specific to the ctypes backend. When creating a ffi.gc wrapper around a pointer cdata, the gc wrapper does not implicitly convert to the type of the wrapped pointer, in function calls and such. Instead the conversion fails with a nonsensical TypeError: cannot convert 'foo *' to 'foo *'.

Short example - this works with the default native backend:

In [1]: import cffi
In [2]: ffi = cffi.FFI()
In [3]: ffi.cdef("typedef struct foo *foo_t; int printf(char *, foo_t);")
In [4]: ptr = ffi.cast("foo_t", 42)
In [5]: gcptr = ffi.gc(ptr, print)
In [6]: libc = ffi.dlopen(None)
In [7]: libc.printf(b"%p\n", ptr)
0x2a
Out[7]: 5

In [8]: libc.printf(b"%p\n", gcptr)
0x2a
Out[8]: 5

But it breaks with the ctypes backend:

In [1]: import cffi
In [2]: import cffi.backend_ctypes
In [3]: ffi = cffi.FFI(backend=cffi.backend_ctypes.CTypesBackend())
In [4]: ffi.cdef("typedef struct foo *foo_t; int printf(char *, foo_t);")
In [5]: ptr = ffi.cast("foo_t", 42)
In [6]: gcptr = ffi.gc(ptr, print)
In [7]: libc = ffi.dlopen(None)
In [8]: libc.printf(b"%p\n", ptr)
0x2a
Out[8]: 5

In [9]: libc.printf(b"%p\n", gcptr)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-7ae6019ad972> in <module>()
----> 1 libc.printf(b"%p\n", gcptr)

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/cffi/backend_ctypes.py in __call__(self, *args)
--> 921                     ctypes_args.append(BArg._arg_to_ctypes(arg))

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/cffi/backend_ctypes.py in _arg_to_ctypes(cls, *value)
---> 39             res = cls._to_ctypes(*value)

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/cffi/backend_ctypes.py in _to_ctypes(cls, value)
--> 214         address = value._convert_to_address(cls)

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/cffi/backend_ctypes.py in _convert_to_address(self, BClass)
--> 232             return CTypesData._convert_to_address(self, BClass)

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/cffi/backend_ctypes.py in _convert_to_address(self, BClass)
---> 88                 self._get_c_name(), BClass._get_c_name()))

TypeError: cannot convert 'struct foo *' to 'struct foo *'

I looked into this a little, and if I'm not mistaken, the issue is in CTypesGenericPtr._convert_to_address:

    def _convert_to_address(self, BClass):
        if (BClass in (self.__class__, None) or BClass._automatic_casts
            or self._automatic_casts):
            return self._address
        else:
            return CTypesData._convert_to_address(self, BClass)

The check if BClass in (self.__class__, None) works with regular pointer cdata, as their class matches the expected parameter type exactly. But the ctypes backend's implementation of gc creates a subclass of the wrapped cdata's type to create the wrapper, and that wrapper subclass is never identical to the expected parameter type (it's only a subclass of the expected type).

CTypesBaseStructOrUnion._convert_to_address looks like it may have the same issue, it does an is check against self.__class__ as well.

Comments (9)

  1. dgelessus reporter

    I didn't know the ctypes backend was that bad... It was a nice alternative when compiling custom extensions isn't an option. I can't really say anything about how well it works with set_source and compile (if I could call a C compiler, I wouldn't be using the ctypes backend...) but it works really well as an interface to ctypes. But I understand that the ctypes backend isn't exactly CFFI's main focus, and it was never documented officially, so I can't really blame you for dropping it.

  2. Armin Rigo

    The only reason to use it would be if you have a CPython pre-installed and no way to install a 3rd-party C module, not even in binary form (there are binaries of cffi on PyPI, at least for Windows and OS/X). Can you explain your use case a bit more? If you convince me, I'll keep the ctypes backend, because it's not that much work to maintain even if it is not working perfectly.

  3. dgelessus reporter

    Yes, my use case is quite niche... There is an iOS app called Pythonista, which is a Python editor/runtime for iOS. (I am not the developer, just a user.) iOS restrictions require that native libraries must be signed by Apple or the app developer, so users cannot compile C extensions themselves. ctypes is supported though and can be used to call native iOS APIs from Python. I've been using CFFI with the ctypes backend here, because it's a lot less work to put the declarations from iOS headers into a cdef than to rewrite everything to ctypes syntax.

    It's your decision of course if you want to support the ctypes backend for odd use cases like these. For Windows and Mac there are wheels on PyPI, and almost every *nix has a C compiler, but it's still nice to have the ctypes backend if all else fails.

  4. Armin Rigo

    Ok, that convinces me. There are other corner cases that won't work with the ctypes backend; please report them here if you can't easily find workarounds. The problem should be fixed in 5fa1d8697d3e and b81ca61b6de6. Can you try it out?

    The real issue was that ffi.typeof(x) is just type(x) for the ctypes backend, so gcp() should not return a subclass. It makes other things not really work. Solved by refactoring a bit gcp().

  5. dgelessus reporter

    OK, thank you for keeping ctypes support :)

    The fix doesn't quite work for me though:

    In [1]: import cffi
       ...: import cffi.backend_ctypes
       ...: ffi = cffi.FFI(backend=cffi.backend_ctypes.CTypesBackend())
       ...: ffi.cdef("typedef struct foo *foo_t; int printf(char *, foo_t);")
       ...: ptr = ffi.cast("foo_t", 42)
       ...: gcptr = ffi.gc(ptr, print)
       ...: libc = ffi.dlopen(None)
       ...: libc.printf(b"%p\n", ptr)
       ...: libc.printf(b"%p\n", gcptr)
       ...: 
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    <ipython-input-1-137d07291fdd> in <module>()
    ----> 6 gcptr = ffi.gc(ptr, print)
    
    /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/cffi/api.py in gc(self, cdata, destructor)
    --> 400         return self._backend.gcp(cdata, destructor)
    
    /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/cffi/backend_ctypes.py in gcp(self, cdata, destructor)
    -> 1030         weak_cache[MyRef(new_cdata, remove)] = (cdata, destructor)
    
    TypeError: unhashable type: 'MyRef'
    
  6. dgelessus reporter

    Looks like it's working now:

    In [1]: import cffi
       ...: import cffi.backend_ctypes
       ...: ffi = cffi.FFI(backend=cffi.backend_ctypes.CTypesBackend())
       ...: ffi.cdef("typedef struct foo *foo_t; int printf(char *, foo_t);")
       ...: ptr = ffi.cast("foo_t", 42)
       ...: gcptr = ffi.gc(ptr, print)
       ...: libc = ffi.dlopen(None)
       ...: libc.printf(b"%p\n", ptr)
       ...: libc.printf(b"%p\n", gcptr)
       ...: 
    0x2a
    0x2a
    Out[1]: 5
    
    In [2]: exit()
    <cdata 'struct foo *' 0x2a>
    

    Thanks for the quick fix!

  7. Log in to comment