Issues

Issue #10 resolved

Python 3.2: GCC warning message leads to UnicodeDecodeError, and Tox stops with InvocationError.

phn
created an issue

Hello,

Tox 1.1 fails with UnicodeDecodeError, when GCC throws a warning message. For example:

"src/tpm/delta_UT.c:15:14: warning: ‘rcsid’ defined but not used".

I can run pip-3.2 (versions 1.0.1 and 1.0.2) outside Tox and the same warning message doesn't cause any trouble.

The same error occurs when using Tox installed on Python 2.6 and on Python 3.2.

I am attaching the last few lines from the Tox log.

Thanks,

Prasanth

Last several lines from tox/log/1.log:

gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tmp/pip-sienco-build/src/tpm -I/home/phn/bin/python32/include/python3.2m -c src/tpm/delta_UT.c -o build/temp.linux-i686-3.2/src/tpm/delta_UT.o Exception: Traceback (most recent call last): File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/basecommand.py", line 126, in main self.run(options, args) File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/commands/install.py", line 228, in run requirement_set.install(install_options, global_options) File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/req.py", line 1100, in install requirement.install(install_options, global_options) File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/req.py", line 572, in install cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False) File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/init.py", line 230, in call_subprocess line = console_to_str(stdout.readline()) File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/backwardcompat.py", line 60, in console_to_str return s.decode(console_encoding) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 35: ordinal not in range(128)

Comments (11)

  1. Marc Abramowitz

    I've seen this too, but I think the error is in pip, as evidenced by the stack trace:

    $ tox -e py31
    ...
      Running setup.py install for phpserialize
        
      Running setup.py install for PyYAML
        checking if libyaml is compilable
        Compiling with an SDK that doesn't seem to exist: /Developer/SDKs/MacOSX10.4u.sdk
        Please check your Xcode installation
        gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/3.1/include/python3.1 -c build/temp.macosx-10.3-fat-3.1/check_libyaml.c -o build/temp.macosx-10.3-fat-3.1/check_libyaml.o
        build/temp.macosx-10.3-fat-3.1/check_libyaml.c:2:18: error: yaml.h: No such file or directory
    Exception:
    Traceback (most recent call last):
      File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/basecommand.py", line 126, in main
        self.run(options, args)
      File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/commands/install.py", line 228, in run
        requirement_set.install(install_options, global_options)
      File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/req.py", line 1100, in install
        requirement.install(install_options, global_options)
      File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/req.py", line 572, in install
        cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
      File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/__init__.py", line 230, in call_subprocess
        line = console_to_str(stdout.readline())
      File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/backwardcompat.py", line 60, in console_to_str
        return s.decode(console_encoding)
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 60: ordinal not in range(128)
    
    Storing complete log in /Users/marc/.pip/pip.log
    
    [TOX] ERROR: could not install deps [pytest, phpserialize, PyYAML]
    

    Note that the exception occurs in pip and tox handles it by displaying a reasonable error message. I guess the error message could be slightly better if it stated that pip install (and the version of pip) is what failed.

    Not sure why this happened for you with tox but not with pip directly -- perhaps tox is using an old version of pip? Duplicate of https://bitbucket.org/hpk42/tox/issue/15/update-to-distribute-0621 perhaps?

  2. Marc Abramowitz

    On second thought and after looking at http://stackoverflow.com/questions/4545661/unicodedecodeerror-when-redirecting-to-file/4551045#4551045, this might not be a pip problem per se.

    The difference might be that pip writing to a terminal works fine but pip writing to a file or pipe does not because Python 2.x can guess the encoding of the terminal, but assumes an ASCII encoding for files and pipes. Tox is calling pip in a subprocess which makes it write to a pipe.

    Doing:

    $ tox --recreate -e py31
    ...
      File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/backwardcompat.py", line 60, in console_to_str
        return s.decode(console_encoding)
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 60: ordinal not in range(128)
    

    But:

    $ PYTHONIOENCODING=utf_8 tox --recreate -e py31
    ...
    [TOX] py31: commands succeeded
    [TOX] congratulations :)
    

    works just fine for me.

    Now the questions are:

    1. Does this not happen with a newer version of pip? Perhaps, they added code to explicitly encode their output to UTF-8?
    2. If 1 is not True, then presumably tox could be modified to set PYTHONIOENCODING when it calls pip?
  3. Log in to comment