Enable sane packaging for cffi applications

Issue #109 resolved
Donald Stufft
created an issue

Currently cffi does compilation on import. This is awesome during development but it's anathema for properly packaged software. It nominally supports being built with distutils/setuptools however to making it actually do this is non obvious.


  • Compilation on import hides installation problems on typical development setups
  • Building with distutils places the built .so not inside of the __pycache__ but alongisde the normal .py files, which cffi doesn't notice and then attempts to compile on import.
  • Putting the compile options into the package code instead of the build script (setup.py) means that developers have to invent their own method of shuttling those configs into the package at install time.
  • I'm pretty sure i've run into another problem but I can't think of it off the top of my head.

Essentially you can't rely on compile on import on production systems because often times you install the package under a different user than what you first execute it as, which won't compile the .so. When you then import it it tries to compile the .so and fails due to permissions errors.

I'm not entirely sure what the best path forward is right now, certainly a good first step would be to ensure that the output of setup.py build is picked up by cffi and then cffi won't try to recompile it.

Comments (30)

  1. Armin Rigo

    Yes, I agree with you completely :-) It is planned to change for the CFFI 1.0 release. (We have in mind a different approach with an explicit compilation step rather than automatically do it. More details to follow)

  2. Donald Stufft reporter

    So the big problem will be finding a place to hook the explicit compilation step into.

    After digging into the build_ext command there doesn't appear to be a generic place to hook a generated .py file into. So what you'd want to do is subclass distutils.commands.build_ext.build_ext and override the build_extension() command in order to also generate the .py file that the FFIBuilder creates. This will let you create the file as part of the build process. Then you'd want to override the get_outputs() method to additionally include the generated .py file.

    This should allow you to both create the generated Python file and include it in the final output.

    You'll probably want to define a FFIBuilder inside of a setup.py and you'd want some sort of a FFIBuilder().extension or something to pass a distutils Extension object (cffi already has something like this). Alternatively instead of a FFIBuilder you could just create a CFFIExtension object which would replace the FFIBuilder and would be passed to the extensions=[...] option in the setup.py. Possibly the CFFIExtension just is an interface to an FFIBuilder I dunno. This combined with the above overridden build_ext would enable a more distutils like experience (with less new gotchas) and more explicit steps.

    If you need help sorting out the distutils stuff or parsing the above ^ I can of course help!

  3. Donald Stufft reporter

    Oh and just incase anyone doesn't know, the way the distutils system works, when you run setup.py install it actually runs one or more build_* commands.

    1. build_clib, builds any c libraries needed for an extension module, doesn't install them, builds them as a .a for static linking.
    2. build_ext, builds extension modules, has any libraries built in the first step available to it.
    3. build_py, basically just a collection option for the packages/py_modules.

    It collects all the outputs of the above into a temporary location, and then the install command queries each command it ran for a list of outputs and then copies them into the final destination.

  4. Vincent Bernat

    OK, the installed .so misses the arch name. Once it has been renamed and moved to the correct location along with the .c file, there is no attempted compilation. I wonder if this is possible to do this in setup.py.

  5. mattip

    build_ext actually builds a python extension library (.pyd on windows). I could not find a way to get distutils to build a non-python pure c shared object (.dll on windows). It would be nice if cffi's shared objects did not need Python.h.

    I would also like to be able to supply a list of exported functions, since by default MSVC does not export functions from a shared object.

  6. Antoine Bertin

    I package for Synology NAS and recently updated pyOpenSSL to 0.14 which requires cryptography which requires cffi. So here I am.

    >>> import OpenSSL
    unable to execute '/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc': No such file or directory
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/local/python/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
        from OpenSSL import rand, crypto, SSL
      File "/usr/local/python/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in <module>
        from OpenSSL._util import (
      File "/usr/local/python/lib/python2.7/site-packages/OpenSSL/_util.py", line 4, in <module>
        binding = Binding()
      File "/usr/local/python/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 89, in __init__
      File "/usr/local/python/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 109, in _ensure_ffi_initialized
      File "/usr/local/python/lib/python2.7/site-packages/cryptography/hazmat/bindings/utils.py", line 80, in build_ffi
      File "/usr/local/python/lib/python2.7/site-packages/cffi/api.py", line 340, in verify
        lib = self.verifier.load_library()
      File "/usr/local/python/lib/python2.7/site-packages/cffi/verifier.py", line 74, in load_library
      File "/usr/local/python/lib/python2.7/site-packages/cffi/verifier.py", line 139, in _compile_module
        outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
      File "/usr/local/python/lib/python2.7/site-packages/cffi/ffiplatform.py", line 25, in compile
        outputfilename = _build(tmpdir, ext)
      File "/usr/local/python/lib/python2.7/site-packages/cffi/ffiplatform.py", line 51, in _build
        raise VerificationError('%s: %s' % (e.__class__.__name__, e))
    cffi.ffiplatform.VerificationError: CompileError: command '/home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc' failed with exit status 1

    I've packaged a lot of python modules with C-extensions but this one is really a PITA. Is there a fix already for this in a branch or PR?

  7. Vincent Bernat

    This issue is becoming quite old and we still don't have a solution. Donald Stufft wrote a bit about it a few months ago: https://caremad.io/2014/11/distributing-a-cffi-project/. More and more people are using projects using CFFI and we now have cross compiling issues where CFFI tries to import the module it built. PyOpenSSL is an example and I have also the same problem in my own project.

    Currently, each project is trying to come with its own hacks. I didn't test all what Donald is explaining on its post but even if it works for crosscompile, this would be a pitta to have such many lines of code just to be able to compile.

    @Donald Stufft, would it be difficult to integrate what you propose in your post directly in CFFI? I suppose that there is some difficulty otherwise you would have proposed the changes in a PR.

  8. Donald Stufft reporter

    Some parts of my blog could be integrates into CFFI, some couldn't. I think some already has been. I haven't had time to figure out which is which and a good way to implement them for a PR.

  9. Donald Stufft reporter

    To be more specific:

    • CFFI could ship with setuptools integration so that a setup.py doesn't need to write install and build commands.
    • A better auto generated module name might already be done, but it shouldn't be hard to do. The biggest thing here is that CFFI should do something like semver for it's ABI so that people can constrain themselves to only ABI compatible versions of CFFI.
    • CFFI could return a LazyLibrary instead of a Library instance as the result of the FFI().verify() call.
    • CFFI could add a parameter to FFI().verify() that disables the implicit compilation (or a different function that'll only do verify + lazy library load, not verify + compile + immediate load).
    • The "better" setup.py is a setuptools problem not a cffi problem which stems from the use of setup_requires.

    Which of these the CFFI project wants to tackle, especially given that they are going to re-architecture CFFI for 1.0 so that it solves all of those issues better, I don't know.

  10. anatoly techtonik

    Mixing build tool with package management is a very bad way to go IMHO.

    As to how to fix this? Every complicated solution should begin with a graph or picture of the problem. Without this I can only guess that the problem is in CFFI trying to figure out which library to load for which system. Is that is the case, why not follow a wheels way and use suffixes for naming and searching for compiled binary code?

    I hope that 1.0 will make that compilation step more explicit and include a diagram or animation of what happens, because if you don't know C - it is all a mystery.

  11. Armin Rigo

    @Donald Stufft: the CFFI 1.0 re-architecture is really an on-and-off project. Depending on it being done in any time frame seems like a bad idea. I would certainly accept any pull request that improves things in the existing cffi. The problem so far has been of the kind: I don't have myself any of the problems of distributing complex cffi modules, so I don't really know for sure what is needed. Someone else should really go ahead and improve the cffi situation about that.

  12. Pierre Vigneras

    FYI, in our case, we prefer the ABI access: we install our .so, and just use cdef()/dlopen(). This leads to some problems too, but we find it simpler from a packaging point of view. We actually generate special python files that defines a constant CDEF containing a string holding the content of a specially generated .h file from the whole batch of .h files we have in our library (just calling gcc -E makes the trick).

    Hope it helps. Regards.

  13. Log in to comment