Use extension modules without building them

Issue #53 closed
g2p
created an issue

I have an application that includes a module that is built from a source checkout. When installed, the app doesn't know the source dir (which will have disappeared in the case of a pip install), and autobuilding in ffi.verify is undesirable (not just because of build products clutter, but because building will generally fail).

To make CFFI modules usable after installation would require separating building the modules from loading them. For example:

#platform.py
ffi = cffi.FFI()
VERIFIER_C = '''
#include <myproj/platform.h>'''
def ext_builder(src_dir):
    return ffi.build(VERIFIER_C, ext_package='platform', include_dirs=[src_dir])
lib = ffi.load(ext_package='platform', builder=ext_builder, autobuild=False)
#setup.py
    ext_modules=[platform.ffi.verifier.get_extension()]

ffi.verifier.get_extension would call the builder registered in ffi.load. Without autobuild=True, the builder should never be called outside of ffi.verifier.get_extension.

(putting this as a proposal, but the current behaviour is a bug that makes distribution of my application difficult)

Comments (5)

  1. Armin Rigo

    Issue 52 was about essentially the same problem, fixed in c2d0ff47f54d using a different solution: forcing the module name instead of relying on _cffi__<hash>. Does this sound acceptable for your application too?

  2. g2p reporter

    That should be workable, with a bit of wrapping so that setup.py clears the right module and forces the rebuild. I'll give it a try.

  3. g2p reporter

    I didn't want to break autorebuild in dev mode, so I went with a solution that didn't pass an explicit modulename but processed the source files at build time so that the varying (cwd-derived) arguments to verify were frozen in the installed modules.

    Freezing modulenames completely (both install time and develop time) would work, but be cumbersome because it would require cleaning built files whenever a parameter changes in dev mode.

    Freezing modulenames at install time to a different value than the CFFI-generated one proved difficult, because the ext_modules mechanism uses imports rather than an explicit verify() call, and it's hard to special-case the verify arguments at this stage. It would be ideal if it could be made to work in a non-intrusive manner.

    tl;dr modulename doesn't simplify my use case. It is possible to make things work on a case by case basis (using distutils magic to freeze verify arguments that used to vary). Making things work better requires deeper changes.

  4. g2p reporter

    paniq's other idea in #52 isn't bad though, as it would ensure that any hacks to keep the hash static work properly or fail early:

    The verifier should also support failing with an exception when it can't find the module instead of trying to build it anew

  5. Log in to comment