AssertionError: version mismatch

Issue #298 resolved
Mike Jarvis
created an issue

I have a project (TreeCorr) that uses cffi. On one of my machines (CentOS Linux release 7.2.1511) I get an error at runtime when I import treecorr after pip installing it:

>>> import treecorr
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mjarvis/lib/python2.7/site-packages/TreeCorr-3.3.6-py2.7-linux-x86_64.egg/treecorr/__init__.py", line 38, in <module>
    _ffi = cffi.FFI()
  File "/home/mjarvis/.local/lib/python2.7/site-packages/cffi/api.py", line 59, in __init__
    "version mismatch, %s != %s" % (backend.__version__, __version__)
AssertionError: version mismatch, 1.6.0 != 1.9.1

I've found other reports of people hitting this problem, but I didn't see an issue here yet.

The workaround is to pip install the same version the backend uses. In the above case:

$ pip install cffi==1.6.0

However, I would like to not require my users to hit this problem in the first place and have to manually do the above fix.

So do you know of any programmatic way in setup.py that I can check what backend is installed on their machine and have TreeCorr depend on the correct cffi version?

Or, even better, do you think it's possible that cffi can automatically do this itself on systems that are going to have this problem? Maybe have pip install cffi install the correct backend along with the frontend? Or at least detect the backend itself and install the correct version of the frontend? Maybe that's not possible though.

Comments (7)

  1. Armin Rigo

    As far as I know, this means something is really broken in the installation. Both the cffi front-end and backend are part of the same package, cffi. If you install some version of one, you normally install the other at the same time. If you get a mismatch it means something is wrong, like pip (somehow) installed some version of the front-end (the pure Python part) without recompiling the backend (the C part). Maybe there is one version of the backend (here 1.6.0) that was installed differently and that takes precedence over the version that pip tries to install (here 1.9.1).

    I would love to hear more about what the mismatch is caused by, e.g. how pip can be in such a state that "pip install cffi" doesn't fix it.

  2. Mike Jarvis reporter

    I'm not sure where to investigate, but if you tell me what to look for, I can report back.

    Here is a session that shows evidence of the problem, but I don't know if it has the clues you need to diagnose it. It starts with cffi 1.6.0 installed (via pip install cffi==1.6.0 --user).

    $ lsb_release -a
    LSB Version:    :core-4.1-amd64:core-4.1-noarch
    Distributor ID: CentOS
    Description:    CentOS Linux release 7.2.1511 (Core) 
    Release:    7.2.1511
    Codename:   Core
    $ python
    Python 2.7.12 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:42:40) 
    [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    Anaconda is brought to you by Continuum Analytics.
    Please check out: http://continuum.io/thanks and https://anaconda.org
    >>> import cffi
    >>> cffi.__file__
    '/home/mjarvis/.local/lib/python2.7/site-packages/cffi/__init__.pyc'
    >>> import treecorr
    >>> 
    $ ls -laF ~/.local/lib/python2.7/site-packages/cffi
    total 668
    drwxrwxr-x  2 mjarvis  4096 Dec 23 12:41 ./
    drwx------ 35 mjarvis  4096 Dec 23 13:59 ../
    -rw-rw-r--  1 mjarvis 37637 Dec 23 12:41 api.py
    -rw-rw-r--  1 mjarvis 36064 Dec 23 12:41 api.pyc
    -rw-rw-r--  1 mjarvis 40130 Dec 23 12:41 backend_ctypes.py
    -rw-rw-r--  1 mjarvis 44152 Dec 23 12:41 backend_ctypes.pyc
    -rw-rw-r--  1 mjarvis 10013 Dec 23 12:41 _cffi_include.h
    -rw-rw-r--  1 mjarvis  5477 Dec 23 12:41 cffi_opcode.py
    -rw-rw-r--  1 mjarvis  5704 Dec 23 12:41 cffi_opcode.pyc
    -rw-rw-r--  1 mjarvis  2531 Dec 23 12:41 commontypes.py
    -rw-rw-r--  1 mjarvis  2236 Dec 23 12:41 commontypes.pyc
    -rw-rw-r--  1 mjarvis 37065 Dec 23 12:41 cparser.py
    -rw-rw-r--  1 mjarvis 24134 Dec 23 12:41 cparser.pyc
    -rw-rw-r--  1 mjarvis 17249 Dec 23 12:41 _embedding.h
    -rw-rw-r--  1 mjarvis  3729 Dec 23 12:41 ffiplatform.py
    -rw-rw-r--  1 mjarvis  4643 Dec 23 12:41 ffiplatform.pyc
    -rw-rw-r--  1 mjarvis   642 Dec 23 12:41 gc_weakref.py
    -rw-rw-r--  1 mjarvis  1202 Dec 23 12:41 gc_weakref.pyc
    -rw-rw-r--  1 mjarvis   483 Dec 23 12:41 __init__.py
    -rw-rw-r--  1 mjarvis   533 Dec 23 12:41 __init__.pyc
    -rw-rw-r--  1 mjarvis   747 Dec 23 12:41 lock.py
    -rw-rw-r--  1 mjarvis   433 Dec 23 12:41 lock.pyc
    -rw-rw-r--  1 mjarvis 21110 Dec 23 12:41 model.py
    -rw-rw-r--  1 mjarvis 22982 Dec 23 12:41 model.pyc
    -rw-rw-r--  1 mjarvis  5835 Dec 23 12:41 parse_c_type.h
    -rw-rw-r--  1 mjarvis 60709 Dec 23 12:41 recompiler.py
    -rw-rw-r--  1 mjarvis 52726 Dec 23 12:41 recompiler.pyc
    -rw-rw-r--  1 mjarvis  6158 Dec 23 12:41 setuptools_ext.py
    -rw-rw-r--  1 mjarvis  6768 Dec 23 12:41 setuptools_ext.pyc
    -rw-rw-r--  1 mjarvis 41370 Dec 23 12:41 vengine_cpy.py
    -rw-rw-r--  1 mjarvis 37135 Dec 23 12:41 vengine_cpy.pyc
    -rw-rw-r--  1 mjarvis 26633 Dec 23 12:41 vengine_gen.py
    -rw-rw-r--  1 mjarvis 24187 Dec 23 12:41 vengine_gen.pyc
    -rw-rw-r--  1 mjarvis 11518 Dec 23 12:41 verifier.py
    -rw-rw-r--  1 mjarvis 11208 Dec 23 12:41 verifier.pyc
    $ ls -laF ~/.local/lib/python2.7/site-packages/cffi-1.6.0.dist-info/
    total 40
    drwxrwxr-x  2 mjarvis 4096 Dec 23 12:41 ./
    drwx------ 35 mjarvis 4096 Dec 23 13:59 ../
    -rw-rw-r--  1 mjarvis  219 Dec 23 12:41 DESCRIPTION.rst
    -rw-rw-r--  1 mjarvis   76 Dec 23 12:41 entry_points.txt
    -rw-rw-r--  1 mjarvis    4 Dec 23 12:41 INSTALLER
    -rw-rw-r--  1 mjarvis 1038 Dec 23 12:41 METADATA
    -rw-rw-r--  1 mjarvis 1070 Dec 23 12:41 metadata.json
    -rw-rw-r--  1 mjarvis 2428 Dec 23 12:41 RECORD
    -rw-rw-r--  1 mjarvis   19 Dec 23 12:41 top_level.txt
    -rw-rw-r--  1 mjarvis  103 Dec 23 12:41 WHEEL
    $ pip install -U cffi --user
    Collecting cffi
      Using cached cffi-1.9.1-cp27-cp27mu-manylinux1_x86_64.whl
    Requirement already up-to-date: pycparser in ./.local/lib/python2.7/site-packages (from cffi)
    Installing collected packages: cffi
      Found existing installation: cffi 1.6.0
        Uninstalling cffi-1.6.0:
          Successfully uninstalled cffi-1.6.0
    Successfully installed cffi-1.9.1
    You are using pip version 8.1.2, however version 9.0.1 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    $ ls -laF ~/.local/lib/python2.7/site-packages/cffi
    total 672
    drwxrwxr-x  2 mjarvis  4096 Dec 24 10:58 ./
    drwx------ 36 mjarvis  4096 Dec 24 10:58 ../
    -rw-rw-r--  1 mjarvis 37242 Dec 24 10:58 api.py
    -rw-rw-r--  1 mjarvis 35828 Dec 24 10:58 api.pyc
    -rw-rw-r--  1 mjarvis 41745 Dec 24 10:58 backend_ctypes.py
    -rw-rw-r--  1 mjarvis 45940 Dec 24 10:58 backend_ctypes.pyc
    -rw-rw-r--  1 mjarvis 10238 Dec 24 10:58 _cffi_include.h
    -rw-rw-r--  1 mjarvis  5477 Dec 24 10:58 cffi_opcode.py
    -rw-rw-r--  1 mjarvis  5704 Dec 24 10:58 cffi_opcode.pyc
    -rw-rw-r--  1 mjarvis  2678 Dec 24 10:58 commontypes.py
    -rw-rw-r--  1 mjarvis  2345 Dec 24 10:58 commontypes.pyc
    -rw-rw-r--  1 mjarvis 37419 Dec 24 10:58 cparser.py
    -rw-rw-r--  1 mjarvis 24323 Dec 24 10:58 cparser.pyc
    -rw-rw-r--  1 mjarvis 17274 Dec 24 10:58 _embedding.h
    -rw-rw-r--  1 mjarvis  3860 Dec 24 10:58 ffiplatform.py
    -rw-rw-r--  1 mjarvis  4755 Dec 24 10:58 ffiplatform.pyc
    -rw-rw-r--  1 mjarvis   483 Dec 24 10:58 __init__.py
    -rw-rw-r--  1 mjarvis   528 Dec 24 10:58 __init__.pyc
    -rw-rw-r--  1 mjarvis   747 Dec 24 10:58 lock.py
    -rw-rw-r--  1 mjarvis   433 Dec 24 10:58 lock.pyc
    -rw-rw-r--  1 mjarvis 21352 Dec 24 10:58 model.py
    -rw-rw-r--  1 mjarvis 23022 Dec 24 10:58 model.pyc
    -rw-rw-r--  1 mjarvis  5835 Dec 24 10:58 parse_c_type.h
    -rw-rw-r--  1 mjarvis 61394 Dec 24 10:58 recompiler.py
    -rw-rw-r--  1 mjarvis 53381 Dec 24 10:58 recompiler.pyc
    -rw-rw-r--  1 mjarvis  7366 Dec 24 10:58 setuptools_ext.py
    -rw-rw-r--  1 mjarvis  7768 Dec 24 10:58 setuptools_ext.pyc
    -rw-rw-r--  1 mjarvis 41373 Dec 24 10:58 vengine_cpy.py
    -rw-rw-r--  1 mjarvis 37125 Dec 24 10:58 vengine_cpy.pyc
    -rw-rw-r--  1 mjarvis 26633 Dec 24 10:58 vengine_gen.py
    -rw-rw-r--  1 mjarvis 24187 Dec 24 10:58 vengine_gen.pyc
    -rw-rw-r--  1 mjarvis 11518 Dec 24 10:58 verifier.py
    -rw-rw-r--  1 mjarvis 11208 Dec 24 10:58 verifier.pyc
    $ ls -laF ~/.local/lib/python2.7/site-packages/cffi-1.9.1.dist-info/
    total 40
    drwxrwxr-x  2 mjarvis 4096 Dec 24 10:58 ./
    drwx------ 36 mjarvis 4096 Dec 24 10:58 ../
    -rw-rw-r--  1 mjarvis  219 Dec 24 10:58 DESCRIPTION.rst
    -rw-rw-r--  1 mjarvis   76 Dec 24 10:58 entry_points.txt
    -rw-rw-r--  1 mjarvis    4 Dec 24 10:58 INSTALLER
    -rw-rw-r--  1 mjarvis 1088 Dec 24 10:58 METADATA
    -rw-rw-r--  1 mjarvis 1111 Dec 24 10:58 metadata.json
    -rw-rw-r--  1 mjarvis 2433 Dec 24 10:58 RECORD
    -rw-rw-r--  1 mjarvis   19 Dec 24 10:58 top_level.txt
    -rw-rw-r--  1 mjarvis  110 Dec 24 10:58 WHEEL
    $ python
    Python 2.7.12 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:42:40) 
    [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    Anaconda is brought to you by Continuum Analytics.
    Please check out: http://continuum.io/thanks and https://anaconda.org
    >>> import cffi
    >>> cffi.__file__
    '/home/mjarvis/.local/lib/python2.7/site-packages/cffi/__init__.pyc'
    >>> import treecorr
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/mjarvis/lib/python2.7/site-packages/TreeCorr-3.3.6-py2.7-linux-x86_64.egg/treecorr/__init__.py", line 38, in <module>
        _ffi = cffi.FFI()
      File "/home/mjarvis/.local/lib/python2.7/site-packages/cffi/api.py", line 59, in __init__
        "version mismatch, %s != %s" % (backend.__version__, __version__)
    AssertionError: version mismatch, 1.6.0 != 1.9.1
    >>> 
    

    So if looks like all the files in the cffi directory were updated. Time stamps are today. And there is no directory cffi-1.6.0.dist-info/ there anymore after the update. So where else might the cffi backend live? Is there a way I can access the file name and location of whatever backend file is being loaded that is still 1.6.0?

  3. Armin Rigo

    I followed your steps but can't reproduce. The file you should look for is _cffi_backend.so which is not inside the site-packages/cffi directory but outside it. Try to look where the 1.6.0 version of _cffi_backend.so is still found by the last step, by printing import _cffi_backend; _cffi_backend.__file__. Maybe it was put somewhere else manually maybe by some older version of pip or something, and isn't overridden by the new 1.9.1 _cffi_backend.so...

  4. Mike Jarvis reporter

    Thanks! That was exactly what I needed. It turned out I had a _cffi_backend.so sitting in a different directory in my library path that superseded the one in .local/lib/python2.7/site-packages:

    $ python
    Python 2.7.12 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:42:40) 
    [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    Anaconda is brought to you by Continuum Analytics.
    Please check out: http://continuum.io/thanks and https://anaconda.org
    >>> import cffi
    >>> cffi.__version__
    '1.9.1'
    >>> cffi.__file__
    '/home/mjarvis/.local/lib/python2.7/site-packages/cffi/__init__.pyc'
    >>> import _cffi_backend
    >>> _cffi_backend.__version__
    '1.6.0'
    >>> _cffi_backend.__file__
    '/home/mjarvis/lib/python2.7/site-packages/_cffi_backend.so'
    >>> 
    $ ls -l /home/mjarvis/lib/python2.7/site-packages/_cffi_backend.so
    -rwxrwxr-x 1 mjarvis 456698 May 10  2016 /home/mjarvis/lib/python2.7/site-packages/_cffi_backend.so*
    $ ls -l /home/mjarvis/.local/lib/python2.7/site-packages/_cffi_backend.so
    -rwxrwxr-x 1 mjarvis 733736 Dec 24 10:58 /home/mjarvis/.local/lib/python2.7/site-packages/_cffi_backend.so*
    $
    

    I must have previously installed with python setup.py install --prefix=~ rather than using pip. When I deleted that file, it found the correct one in the .local tree, and everything works fine. Thanks for your help!

    I'm not sure if there is anything to be done on your end to make this easier for future users who might run into a similar problem. Maybe add a bit more to the error message like "This shouldn't happen. Check if you have an old _cffi_backend.so file elsewhere in your python path that could be superseding the more recent one, and if so delete it."

    In any case, my specific problem is solved, so thanks again.

  5. Log in to comment