Issue #52 resolved

Ability to fully override naming of temporary files

Leonard Ritter
created an issue

I would like to be able to fully override the naming of temporary files to ensure cffi's verifier has no chance to act up and change the hash for generated modules. The verifier should also support failing with an exception when it can't find the module instead of trying to build it anew.

The current automagical behavior is neat when it works, but can be absolutely unnerving when it doesn't. Less moving parts == less guesswork and awkward debugging sessions for me.

Rationale: I'm occasionally running into weird issues where a cffi-based package is installed with its compiled extension module, but the verifier still insists on recompiling it. In this instance, the error occured with CPython 2.7 (the Portable Python build), although I have also experienced it with PyPy before. I'm passing

ext_modules=[bullet.internal._ffi.verifier.get_extension()]

to the setup() function in setup.py. When the setup.py imports the bullet module, the verifier generates py and pyd modules with the hash prefix "_cffi__x3eb38760xb07eed14", and these are being installed.

However, when first importing the installed module, CFFI expects a different hash:

>>> import bullet
load_lib() called
_cffi__x2d4f5f88x997efa59.c
lib\site-packages\pybullet_cffi-0.1-py2.7-win32.egg\bullet\__pycache__\_cffi__x2
d4f5f88x997efa59.c(151) : fatal error C1083: Cannot open include file: 'bullet_w
rapper.h': No such file or directory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ppy27\lib\site-packages\pybullet_cffi-0.1-py2.7-win32.egg\bullet\__in
it__.py", line 4, in <module>
    from .proxy import *
  File "C:\ppy27\lib\site-packages\pybullet_cffi-0.1-py2.7-win32.egg\bullet\prox
y.py", line 7, in <module>
    from ._bullet import *
  File "C:\ppy27\lib\site-packages\pybullet_cffi-0.1-py2.7-win32.egg\bullet\_bul
let.py", line 523, in <module>
    """)
  File "C:\ppy27\lib\site-packages\pybullet_cffi-0.1-py2.7-win32.egg\bullet\inte
rnal.py", line 84, in load_lib
    _LIB = _ffi.verify(VERIFY_SOURCE, **VERIFY_OPTIONS)
  File "C:\ppy27\lib\site-packages\cffi-0.5-py2.7-win32.egg\cffi\api.py", line 2
99, in verify
    lib = self.verifier.load_library()
  File "C:\ppy27\lib\site-packages\cffi-0.5-py2.7-win32.egg\cffi\verifier.py", l
ine 63, in load_library
    self.compile_module()
  File "C:\ppy27\lib\site-packages\cffi-0.5-py2.7-win32.egg\cffi\verifier.py", l
ine 51, in compile_module
    self._compile_module()
  File "C:\ppy27\lib\site-packages\cffi-0.5-py2.7-win32.egg\cffi\verifier.py", l
ine 132, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "C:\ppy27\lib\site-packages\cffi-0.5-py2.7-win32.egg\cffi\ffiplatform.py"
, line 25, in compile
    outputfilename = _build(tmpdir, ext)
  File "C:\ppy27\lib\site-packages\cffi-0.5-py2.7-win32.egg\cffi\ffiplatform.py"
, line 50, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.ffiplatform.VerificationError: CompileError: command '"c:\Program Files (x8
6)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe"' failed with exit status 2

I have given up on hunting down these issues, I just want it to stop. Stuff like this makes CFFI appear fragile and moody instead of cunningly clever. With a more primitive alternative, a module could be outdated, but this is easier to remedy (just delete all generated files) than this autoelectric hellhole.

Comments (5)

  1. Armin Rigo

    I see your point, I'll think about it. Note also that you can try to debug why the hash is different, by printing the string out of which the hash is computed, and comparing. ('key', in cffi/verifier.py line 18).

  2. Leonard Ritter reporter

    This is the beginning of the key during setup.py:

    '\'2.7\\x000.5\\x00\\n#include "bullet_wrapper.h"\\n\\x006d18sextra_com
    pile_args0l15sextra_link_args4l56sc:\\\\devel\\\\pybullet-cffi\\\\bullet-src\\\\
    lib\\\\BulletSoftBody.lib56sc:\\\\devel\\\\pybullet-cffi\\\\bullet-src\\\\lib\\\
    \BulletDynamics.lib57sc:\\\\devel\\\\pybullet-cffi\\\\bullet-src\\\\lib\\\\Bulle
    tCollision.lib52sc:\\\\devel\\\\pybullet-cffi\\\\bullet-src\\\\lib\\\\LinearMath
    .lib12sinclude_dirs2l27sc:\\\\devel\\\\pybullet-cffi\\\\defs37sc:\\\\devel\\\\py
    bullet-cffi\\\\bullet-src\\\\src9slibraries0l12slibrary_dirs1l37sc:\\\\devel\\\\
    pybullet-cffi\\\\bullet-src\\\\lib7ssources1l46sc:\\\\devel\\\\pybullet-cffi\\\\
    defs\\\\bullet_wrapper.cpp\\x00\\ntypedef struct _btVector3 btVector3;\\ntypedef
    

    But this is what it looks like when the module is installed:

    '\'2.7\\x000.5\\x00\\n#include "bullet_wrapper.h"\\n\\x006d18sextra_com
    pile_args0l15sextra_link_args4l94sC:\\\\ppy27\\\\lib\\\\site-packages\\\\pybulle
    t_cffi-0.1-py2.7-win32.egg\\\\bullet-src\\\\lib\\\\BulletSoftBody.lib94sC:\\\\pp
    y27\\\\lib\\\\site-packages\\\\pybullet_cffi-0.1-py2.7-win32.egg\\\\bullet-src\\
    \\lib\\\\BulletDynamics.lib95sC:\\\\ppy27\\\\lib\\\\site-packages\\\\pybullet_cf
    fi-0.1-py2.7-win32.egg\\\\bullet-src\\\\lib\\\\BulletCollision.lib90sC:\\\\ppy27
    \\\\lib\\\\site-packages\\\\pybullet_cffi-0.1-py2.7-win32.egg\\\\bullet-src\\\\l
    ib\\\\LinearMath.lib12sinclude_dirs2l65sC:\\\\ppy27\\\\lib\\\\site-packages\\\\p
    ybullet_cffi-0.1-py2.7-win32.egg\\\\defs75sC:\\\\ppy27\\\\lib\\\\site-packages\\
    \\pybullet_cffi-0.1-py2.7-win32.egg\\\\bullet-src\\\\src9slibraries0l12slibrary_
    dirs1l75sC:\\\\ppy27\\\\lib\\\\site-packages\\\\pybullet_cffi-0.1-py2.7-win32.eg
    g\\\\bullet-src\\\\lib7ssources1l84sC:\\\\ppy27\\\\lib\\\\site-packages\\\\pybul
    let_cffi-0.1-py2.7-win32.egg\\\\defs\\\\bullet_wrapper.cpp\\x00\\ntypedef struct
    

    As you can see, the keyword arguments passed to verify() are wildly different because of differing paths, which are collated like this:

    THISDIR = os.path.abspath(os.path.dirname(__file__))
    CDEF_PATH = os.path.abspath(os.path.join(THISDIR, '..', 'defs'))
    
    if sys.platform == 'win32':
        HEADER_PATH = os.path.abspath(os.path.join(THISDIR, '..', 'bullet-src', 'src'))
        LIB_PATH = os.path.abspath(os.path.join(THISDIR, '..', 'bullet-src', 'lib'))
        INCLUDE_PATH = HEADER_PATH
    else:
        HEADER_PATH = os.path.abspath(os.path.join(THISDIR, '..', 'bullet-bin', 'include'))
        LIB_PATH = os.path.abspath(os.path.join(THISDIR, '..', 'bullet-bin', 'lib'))
        INCLUDE_PATH = os.path.join(HEADER_PATH, 'bullet')
    HEADER_NAME = 'btBulletDynamicsCommon.h'
    ROOT_HEADER_PATH = os.path.join(INCLUDE_PATH, HEADER_NAME)
    LIBNAMES = [
    'BulletSoftBody',
    'BulletDynamics',
    'BulletCollision',
    'LinearMath'
    ]
    if sys.platform == 'win32':
        LINK_ARGS = [
        ]
        LIBS = []
        STATIC_LIBS = [os.path.join(LIB_PATH, lib + '.lib') for lib in LIBNAMES]
    else: 
        LINK_ARGS = [
            '-fPIC',
        ]
        LIBS = [
            'stdc++',
        ]
        STATIC_LIBS = [os.path.join(LIB_PATH, 'lib' + lib + '.a') for lib in LIBNAMES]
    VERIFY_SOURCE = """
    #include "bullet_wrapper.h"
    """
    VERIFY_OPTIONS = dict(
        sources = [
            os.path.join(CDEF_PATH, 'bullet_wrapper.cpp')            
        ],
        extra_link_args = LINK_ARGS + STATIC_LIBS,
        libraries = LIBS,
        library_dirs = [
            LIB_PATH
        ],
        include_dirs = [
            CDEF_PATH,
            INCLUDE_PATH,
        ],
        extra_compile_args = [
        ],
    )
    

    meaning the key will always differ when absolute paths are calculated from relative locations.

  3. Log in to comment