segfault while installing via pip

Create issue
Issue #272 closed
Julien Palard created an issue

All informations are here: http://bugs.python.org/issue27542 I have a coredump if it's of any help but it should be easy to reproduce with the same versions.

Comments (12)

  1. Armin Rigo

    So just to confirm, you're getting a segfault installing cffi with a Linux binary wheel but not from source? That's just another reason for why I'm -0 on Linux binary wheels. If you can confirm, then I will remove the Linux binary wheels from https://pypi.python.org/pypi/cffi and ask that the contributor stops uploading them there until he has got an idea about the problem you describe.

  2. reaperhulk

    Yikes, yeah if that's happening with the manylinux1 wheel and not when compiling from source (and that issue seems to say that) then we should definitely delete the wheels.

  3. Julien Palard reporter

    Happening with python -m pip install cffi, but not with python -m pip install --no-use-wheel cffi yes. Look like the segfault happen while exiting the main python interpreter, which looks like a simple refcount error (look at my bt), but I can get this wrong. I tried gdb on the core dump, but that was useless for me: It's in the subtract_refs of gcmodule.c, only iterating a very long chained list of items to destroy, problematic object probably simply in the list, even if I find it I won't learn anything from it. I tried strace, but just to confirm it happen very late in the process: while exiting, after a close(4) and after a close(3): very late.

    So It's a refcount error, if I can be of any help finding it, tell me.

    FYI after testing a few things (like compiling a python with debug flags) it does no longer happen, did you removed the binary wheel while I as testing?

  4. Julien Palard reporter

    Ok so I chanced "something" in my environment and I'm no longer to reproduce it, same version of pip and same version of python, I had cffi previously installed via apt, I dropped it an reinstalled it while testing. So it may have been upgraded during this manipulation. Other than that I don't know what changed. Still have the core dump if it can help.

  5. Donald Stufft

    Debuntu ships CFFI backend separately from the metadata that says cffi is installed. That might have been confusing things.

  6. Armin Rigo

    Note that unless I'm being confused, you got once:

    $ python -m pip install --user cffi
    Collecting cffi
      Using cached cffi-1.7.0-cp27-cp27mu-manylinux1_x86_64.whl
    Collecting pycparser (from cffi)
    Installing collected packages: pycparser, cffi
    Successfully installed cffi-1.7.0 pycparser-2.14
    Segmentation fault (core dumped)
    

    This should only have unpacked the wheel and put it in the correct paths, not actually run anything from it. In particular, the _cffi_backend C extension module is not imported at all until someone really instantiate a cffi.FFI() object, so even if the pip logic tries to import the cffi module to look around some version number, it should not import _cffi_backend. This would seem to mean that the problem was likely inside CPython or pip itself.

  7. Julien Palard reporter

    I agree, that's why I opened it on bugs.python.org first, I'm reopening the python bug to take this into consideration.

  8. Thomas Meyssonnier

    I've had this again, and it possibly has to do with cffi because it appears in my pip install and because googleing this specific error led me here. But still, it's only cffi triggering a bug in pip, which definitely shouldn't segfault even with a nonsensical package. Please, could someone make a proper bug report against pip ? I personnaly don't have the skills to do this. Log attached, any extra information can be made available if necessary. Thanks.

    user@host:~$ python -m pip install --user -r $IDF_PATH/requirements.txt
    Collecting setuptools (from -r /home/user/esp/esp-idf/requirements.txt (line 4))
      Using cached https://files.pythonhosted.org/packages/37/06/754589caf971b0d2d48f151c2586f62902d93dc908e2fd9b9b9f6aa3c9dd/setuptools-40.6.3-py2.py3-none-any.whl
    Collecting pyserial>=3.0 (from -r /home/user/esp/esp-idf/requirements.txt (line 8))
      Using cached https://files.pythonhosted.org/packages/0d/e4/2a744dd9e3be04a0c0907414e2a01a7c88bb3915cbe3c8cc06e209f59c30/pyserial-3.4-py2.py3-none-any.whl
    Collecting future>=0.15.2 (from -r /home/user/esp/esp-idf/requirements.txt (line 9))
    Collecting cryptography>=2.1.4 (from -r /home/user/esp/esp-idf/requirements.txt (line 10))
      Using cached https://files.pythonhosted.org/packages/7f/ba/383b51cc26e3141c689ce988814385c7659f5ba01c4b5f2de38233010b5f/cryptography-2.4.2-cp27-cp27mu-manylinux1_x86_64.whl
    Collecting pyparsing>=2.0.3 (from -r /home/user/esp/esp-idf/requirements.txt (line 11))
      Using cached https://files.pythonhosted.org/packages/de/0a/001be530836743d8be6c2d85069f46fecf84ac6c18c7f5fb8125ee11d854/pyparsing-2.3.1-py2.py3-none-any.whl
    Collecting enum34; python_version < "3" (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
    Collecting idna>=2.1 (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
    Collecting asn1crypto>=0.21.0 (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
    Collecting cffi!=1.11.3,>=1.7 (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Using cached https://files.pythonhosted.org/packages/14/dd/3e7a1e1280e7d767bd3fa15791759c91ec19058ebe31217fe66f3e9a8c49/cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl
    Collecting six>=1.4.1 (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
    Collecting ipaddress; python_version < "3" (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
    Collecting pycparser (from cffi!=1.11.3,>=1.7->cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
    Installing collected packages: setuptools, pyserial, future, enum34, idna, asn1crypto, pycparser, cffi, six, ipaddress, cryptography, pyparsing
    Successfully installed asn1crypto-0.24.0 cffi-1.11.5 cryptography-2.4.2 enum34-1.1.6 future-0.17.1 idna-2.8 ipaddress-1.0.22 pycparser-2.19 pyparsing-2.3.1 pyserial-3.4 setuptools-40.6.3 six-1.12.0
    Segmentation fault (core dumped)
    user@host:~$ python -m pip install --no-use-wheel --user -r $IDF_PATH/requirements.txt
    DEPRECATION: --no-use-wheel is deprecated and will be removed in the future.  Please use --no-binary :all: instead.
    Collecting setuptools (from -r /home/user/esp/esp-idf/requirements.txt (line 4))
      Downloading https://files.pythonhosted.org/packages/37/1b/b25507861991beeade31473868463dad0e58b1978c209de27384ae541b0b/setuptools-40.6.3.zip (839kB)
        100% |████████████████████████████████| 839kB 231kB/s 
    Collecting pyserial>=3.0 (from -r /home/user/esp/esp-idf/requirements.txt (line 8))
      Downloading https://files.pythonhosted.org/packages/cc/74/11b04703ec416717b247d789103277269d567db575d2fd88f25d9767fe3d/pyserial-3.4.tar.gz (151kB)
        100% |████████████████████████████████| 153kB 265kB/s 
    Collecting future>=0.15.2 (from -r /home/user/esp/esp-idf/requirements.txt (line 9))
      Using cached https://files.pythonhosted.org/packages/90/52/e20466b85000a181e1e144fd8305caf2cf475e2f9674e797b222f8105f5f/future-0.17.1.tar.gz
    Collecting cryptography>=2.1.4 (from -r /home/user/esp/esp-idf/requirements.txt (line 10))
      Downloading https://files.pythonhosted.org/packages/f3/39/d3904df7c56f8654691c4ae1bdb270c1c9220d6da79bd3b1fbad91afd0e1/cryptography-2.4.2.tar.gz (468kB)
        100% |████████████████████████████████| 471kB 233kB/s 
    Collecting pyparsing>=2.0.3 (from -r /home/user/esp/esp-idf/requirements.txt (line 11))
      Downloading https://files.pythonhosted.org/packages/b9/b8/6b32b3e84014148dcd60dd05795e35c2e7f4b72f918616c61fdce83d27fc/pyparsing-2.3.1.tar.gz (596kB)
        100% |████████████████████████████████| 604kB 234kB/s 
    Collecting idna>=2.1 (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Downloading https://files.pythonhosted.org/packages/ad/13/eb56951b6f7950cadb579ca166e448ba77f9d24efc03edd7e55fa57d04b7/idna-2.8.tar.gz (174kB)
        100% |████████████████████████████████| 184kB 292kB/s 
    Collecting asn1crypto>=0.21.0 (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Downloading https://files.pythonhosted.org/packages/fc/f1/8db7daa71f414ddabfa056c4ef792e1461ff655c2ae2928a2b675bfed6b4/asn1crypto-0.24.0.tar.gz (104kB)
        100% |████████████████████████████████| 112kB 253kB/s 
    Collecting six>=1.4.1 (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Downloading https://files.pythonhosted.org/packages/dd/bf/4138e7bfb757de47d1f4b6994648ec67a51efe58fa907c1e11e350cddfca/six-1.12.0.tar.gz
    Collecting cffi!=1.11.3,>=1.7 (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Downloading https://files.pythonhosted.org/packages/e7/a7/4cd50e57cc6f436f1cc3a7e8fa700ff9b8b4d471620629074913e3735fb2/cffi-1.11.5.tar.gz (438kB)
        100% |████████████████████████████████| 440kB 228kB/s 
    Collecting enum34 (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Downloading https://files.pythonhosted.org/packages/bf/3e/31d502c25302814a7c2f1d3959d2a3b3f78e509002ba91aea64993936876/enum34-1.1.6.tar.gz (40kB)
        100% |████████████████████████████████| 40kB 14kB/s 
    Collecting ipaddress (from cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Downloading https://files.pythonhosted.org/packages/97/8d/77b8cedcfbf93676148518036c6b1ce7f8e14bf07e95d7fd4ddcb8cc052f/ipaddress-1.0.22.tar.gz
    Collecting pycparser (from cffi!=1.11.3,>=1.7->cryptography>=2.1.4->-r /home/user/esp/esp-idf/requirements.txt (line 10))
      Using cached https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz
    Skipping bdist_wheel for setuptools, due to binaries being disabled for it.
    Skipping bdist_wheel for pyserial, due to binaries being disabled for it.
    Skipping bdist_wheel for future, due to binaries being disabled for it.
    Skipping bdist_wheel for cryptography, due to binaries being disabled for it.
    Skipping bdist_wheel for pyparsing, due to binaries being disabled for it.
    Skipping bdist_wheel for idna, due to binaries being disabled for it.
    Skipping bdist_wheel for asn1crypto, due to binaries being disabled for it.
    Skipping bdist_wheel for six, due to binaries being disabled for it.
    Skipping bdist_wheel for cffi, due to binaries being disabled for it.
    Skipping bdist_wheel for enum34, due to binaries being disabled for it.
    Skipping bdist_wheel for ipaddress, due to binaries being disabled for it.
    Skipping bdist_wheel for pycparser, due to binaries being disabled for it.
    Installing collected packages: setuptools, pyserial, future, idna, asn1crypto, six, pycparser, cffi, enum34, ipaddress, cryptography, pyparsing
      Running setup.py install for setuptools ... done
      Running setup.py install for pyserial ... done
      Running setup.py install for future ... done
      Running setup.py install for idna ... done
      Running setup.py install for asn1crypto ... done
      Running setup.py install for six ... done
      Running setup.py install for pycparser ... done
      Running setup.py install for cffi ... done
      Running setup.py install for enum34 ... done
      Running setup.py install for ipaddress ... done
      Running setup.py install for cryptography ... done
      Running setup.py install for pyparsing ... done
    Successfully installed asn1crypto-0.24.0 cffi-1.11.5 cryptography-2.4.2 enum34-1.1.6 future-0.17.1 idna-2.8 ipaddress-1.0.22 pycparser-2.19 pyparsing-2.3.1 pyserial-3.4 setuptools-40.6.3 six-1.12.0
    user@host:~$ uname -a
    Linux host 4.13.0-46-lowlatency #51-Ubuntu SMP PREEMPT Tue Jun 12 13:17:41 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
    
  9. Armin Rigo

    Thomas Meyssonnier: this issue is closed and your comment is not going to be noticed by anyone but me. It's the wrong place to ask for someone else to open a bug in another project.

  10. Log in to comment