Issue #4 resolved

problems with importing system modules w/ slpython2.7 coexisting with python2.7

Michael Bravo
created an issue

I have a stock Debian Wheezy system, with standard python 2.7 installed:

# uname -a
Linux ===redacted=== 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux
# cat /etc/debian_version 
7.5

Stackless python was installed via pip; everything was in working order.

However, just recently two new packages were also installed in system via pip: syncless and concurrence. Here's the current output of pip freeze:

# pip freeze
List==1.3.0
Pyrex==0.9.9
Routes==2.0
WebOb==1.4
argparse==1.2.1
concurrence==0.3.1
distribute==0.6.24dev-r0
greenlet==0.4.2
repoze.lru==0.6
stackless-python==10.0
syncless==0.25
wsgiref==0.1.2

With this, both syncless and concurrence work ok with system python, say with standard example from syncless:

python -c 'if 1:
    from syncless import wsgi; import stackless
    wsgi.simple(8080, lambda *args: ["Hello, <b>World<\/b>!\n"])
    stackless.schedule_remove()'

(this runs a simple web server, responds on port 8080)

but with slpython it bombs:

# slpython2.7 -c 'if 1:
    from syncless import wsgi; import stackless
    wsgi.simple(8080, lambda *args: ["Hello, <b>World<\/b>!\n"])
    stackless.schedule_remove()'
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/usr/local/lib/python2.7/dist-packages/syncless-0.25-py2.7-linux-x86_64.egg/syncless/wsgi.py", line 131, in <module>
    import logging
  File "logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref
ImportError: No module named time

here are the outputs of sys.path:

system python:

python -c 'import sys; print sys.path'
['', '/usr/local/lib/python2.7/dist-packages/Pyrex-0.9.9-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/concurrence-0.3.1-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/Routes-2.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/repoze.lru-0.6-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/WebOb-1.4-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/syncless-0.25-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/greenlet-0.4.2-py2.7-linux-x86_64.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/pymodules/python2.7']

slpython:

# slpython2.7 -c 'import sys; print sys.path'
['', '/usr/local/lib/python2.7/dist-packages/Pyrex-0.9.9-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/concurrence-0.3.1-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/Routes-2.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/repoze.lru-0.6-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/WebOb-1.4-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/syncless-0.25-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/greenlet-0.4.2-py2.7-linux-x86_64.egg', '/usr/lib/python27.zip', '/usr/lib/python2.7/stackless-packages', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/pymodules/python2.7']

Comments (12)

  1. Anselm Kruis repo owner

    Thanks for the detailed report.

    I can reproduce the problem, event in a virtual env. A simple test case is

    [test] kruis@goinn /tmp > slpython2.7 -c 'import time'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named time
    

    Analysis

    Debian uses a non standard build configuration and has much more built-in modules:

    System python on Debian Wheezy:

    [test] kruis@goinn /tmp > python -c 'import sys; print sys.builtin_module_names'
    ('__builtin__', '__main__', '_ast', '_bisect', '_codecs', '_collections', '_functools', '_heapq', '_locale', '_random', '_socket', '_sre', '_struct', '_symtable', '_warnings', '_weakref', 'array', 'binascii', 'cPickle', 'cStringIO', 'cmath', 'errno', 'exceptions', 'fcntl', 'gc', 'grp', 'imp', 'itertools', 'marshal', 'math', 'operator', 'posix', 'pwd', 'select', 'signal', 'spwd', 'strop', 'sys', 'syslog', 'thread', 'time', 'unicodedata', 'xxsubtype', 'zipimport', 'zlib')
    

    Stackless

    [test] kruis@goinn /tmp > slpython2.7 -c 'import sys; print sys.builtin_module_names'
    ('__builtin__', '__main__', '_ast', '_codecs', '_sre', '_symtable', '_warnings', '_weakref', 'errno', 'exceptions', 'gc', 'imp', 'marshal', 'posix', 'pwd', 'signal', 'sys', 'thread', 'xxsubtype', 'zipimport')
    

    This is a serious problem. Without the external extension packets (shared-libs) provided by the standard build configuration, all the nasty OS specific details regarding library versions (libc, libz, ...) go into libpython. Either we need specialised Debian builds or we have to exclude Debian and Ubuntu from the list of supported systems.

  2. Michael Bravo reporter

    As it stands, 2.7.6 .deb package there is not installable:

    # dpkg -i stackless2.7_2.7.6-1_amd64.deb 
    dpkg: error processing stackless2.7_2.7.6-1_amd64.deb (--install):
     unable to open file '/var/lib/dpkg/tmp.ci//control': No such file or directory
    Errors were encountered while processing:
     stackless2.7_2.7.6-1_amd64.deb
    

    I'll take a quick look at the .deb innards, but that's what is happening

  3. Anselm Kruis repo owner

    New pip installers are available. Simply reinstall "stackless-python"

    pip install -I stackless-python

    should do the trick.

    Important: the new installer packages compile some modules on Debian based systems. This requires various Debian -dev packages. You probably need python2.7-dev and some of libreadline-dev, libtinfo-dev, libncursesw5-dev, libssl-dev, zlib1g-dev, libexpat1-dev, libbluetooth-dev, libffi-dev, libgdbm-dev

  4. Michael Bravo reporter

    Thank you! It looks like success, at least at the first glance:

    # pip install -I stackless-python
    Downloading/unpacking stackless-python
      Downloading stackless-python-10.0.tar.gz
      Running setup.py egg_info for package stackless-python
    
        Installed /root/build/stackless-python/stackless_installer_C4_linux_x86_64-2.7.3.2-py2.7.egg
    
    Installing collected packages: stackless-python
      Running setup.py install for stackless-python
        /usr/bin/python -c import sys;from pkg_resources import load_entry_point;sys.exit(load_entry_point('stackless_installer_C4_linux_x86_64', 'console_scripts', 'install-stackless')())
        /tmp/stackless_installer_rfOG3T/Modules/mathmodule.c: In function math_fsum:
        /tmp/stackless_installer_rfOG3T/Modules/mathmodule.c:1073:5: warning: passing argument 1 of PyFPE_dummy discards volatile qualifier from pointer target type [enabled by default]
        In file included from /usr/include/python2.7/Python.h:156:0,
                         from /tmp/stackless_installer_rfOG3T/Modules/mathmodule.c:55:
        /usr/include/python2.7/pyfpe.h:134:15: note: expected void * but argument is of type volatile double *
        /tmp/stackless_installer_rfOG3T/Modules/_collectionsmodule.c: In function deque_clearmethod:
        /tmp/stackless_installer_rfOG3T/Modules/_collectionsmodule.c:707:9: warning: variable rv set but not used [-Wunused-but-set-variable]
        /tmp/stackless_installer_rfOG3T/Modules/socketmodule.c: In function socket_gethostbyname_ex:
        /tmp/stackless_installer_rfOG3T/Modules/socketmodule.c:3374:9: warning: variable result set but not used [-Wunused-but-set-variable]
        /tmp/stackless_installer_rfOG3T/Modules/socketmodule.c: In function socket_gethostbyaddr:
        /tmp/stackless_installer_rfOG3T/Modules/socketmodule.c:3450:9: warning: variable result set but not used [-Wunused-but-set-variable]
        building dbm using gdbm
        Stackless installer: install successfully completed
        Installed Stackless Python
    
    Successfully installed stackless-python
    Cleaning up...
    root@dev ~ # slpython2.7 
    Python 2.7.3 Stackless 3.1b3 060516 (default, Jul 17 2014, 18:44:33) 
    [GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import time
    >>> 
    
  5. Michael Bravo reporter

    I'm not sure if this is another symptom of the bug, or even if this is a bug at all, however there is this little problem:

    apparently the stackless build was compiled with no --with-pydebug option

    as a result, when trying to build syncless ( https://code.google.com/p/syncless/ ), it bombs out with stack trace like this (command line that was invoked is "slpython2.7 setup.py build" (sans quotes)):

    running build_ext
    Traceback (most recent call last):
      File "setup.py", line 908, in <module>
        symlink_script_src_dirs=['test', 'benchmark', 'coio_src', 'examples'],
      File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
        dist.run_commands()
      File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
        self.run_command(cmd_name)
      File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python2.7/distutils/dist.py", line 971, in run_command
        cmd_obj.ensure_finalized()
      File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
        self.finalize_options()
      File "/usr/lib/python2.7/stackless-packages/distutils/command/build_ext.py", line 150, in finalize_options
        py_include = sysconfig.get_python_inc()
      File "/usr/lib/python2.7/distutils/sysconfig.py", line 88, in get_python_inc
        return os.path.join(prefix, "include", "python" + get_python_version())+(sys.pydebug and "_d" or "")
    AttributeError: 'module' object has no attribute 'pydebug'
    

    as you can see, error gets thrown by line 88 in regular python library, which invokes sys.pdebug

    I can of course just do an ugly hack and comment out sys.pdebug invocation (which is, to be honest, not obvious at all why it is there in the first place) but perhaps you, as better acquainted with the ecosystem, can discern something better in this situation. I can also make this a new issue (if it IS an issue, and with stackless)

    Otherwise, the original issue here seems to be resolved, and thanks a lot for your help and work.

  6. Anselm Kruis repo owner

    Hi Michael,

    unfortunately, the stackless-python installer does not support debug builds (--with-pydebug).

    Probably I should update the umbrella installer to detect a debug build and issue an error message.

  7. Anselm Kruis repo owner

    Hi Michael,

    sorry, your problem is indeed another instance of the usual Debian problems. Debian adds the boolean sys.pydebug. The boolean is True, if it is a debug build. I'll update the installers.

    Thanks for reporting. Anselm

  8. Anselm Kruis repo owner

    Michael Bravo I updated the installer. Please use

    pip install -I stackless-python
    

    to reinstall. Hopefully that's was the last bug. Unfortunately, I can't test this version thoroughly, because I'm currently at the EuroPython conference and have to work remotely.

  9. Log in to comment