PyPy3 2.4.0 OperationError/LookupError with virtualenv and a PATH with non-ASCII characters

Issue #2300 resolved
Chi Hsuan Yen created an issue

Description:

I'm trying to fix a virtualenv issue (https://github.com/pypa/virtualenv/issues/457). All tests on CPython pass and all on PyPy fails. [1] The PyPy2 test is OK on my machine, so I guess it's an Ubuntu issue. The PyPy3 test fails with an OperationError both on Travis CI and my machine:

debug: OperationError:
debug:  operror-type: LookupError
debug:  operror-value: no codec search functions registered: can't find encoding
debug: OperationError:
debug:  operror-type: AttributeError
debug:  operror-value: stdout

Steps to reproduce:

$ git clone -b encoding-fix https://github.com/yan12125/virtualenv
$ cd virtualenv
$ pypy3 virtualenv.py 中文

The same result appears on Ubuntu 12.04 (Travis CI) [2] and Arch Linux (my machine).

Additional info:

PyPy3 version on my machine:

$ pypy3 -V
Python 3.2.5 (b2091e973da69152b3f928bfaabd5d2347e6df46, Mar 04 2016, 07:08:30)
[PyPy 2.4.0 with GCC 5.3.0]

And on Travis CI:

$ python --version
Python 3.2.5 (b2091e973da6, Oct 19 2014, 18:29:55)
[PyPy 2.4.0 with GCC 4.6.3]

[1] https://travis-ci.org/pypa/virtualenv/builds/130198936

[2] https://travis-ci.org/pypa/virtualenv/jobs/130198943

Comments (5)

  1. Chi Hsuan Yen reporter

    The last few lines from strace -f ./bin/pypy3 in ./中文/:

    stat("/home/yen/tmp/virtualenv/\344\270\255\346\226\207/lib_pypy/__extensions__/encodings", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    stat("/home/yen/tmp/virtualenv/\344\270\255\346\226\207/lib_pypy/__extensions__/encodings.py", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    stat("/home/yen/tmp/virtualenv/\344\270\255\346\226\207/lib_pypy/__extensions__/encodings.pyc", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    stat("/home/yen/tmp/virtualenv/\344\270\255\346\226\207/lib_pypy/__extensions__/encodings.pypy3-24.so", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    stat("/home/yen/tmp/virtualenv/\344\270\255\346\226\207/lib_pypy/encodings", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    stat("/home/yen/tmp/virtualenv/\344\270\255\346\226\207/lib_pypy/encodings.py", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    stat("/home/yen/tmp/virtualenv/\344\270\255\346\226\207/lib_pypy/encodings.pyc", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    stat("/home/yen/tmp/virtualenv/\344\270\255\346\226\207/lib_pypy/encodings.pypy3-24.so", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    stat("/home/yen/tmp/virtualenv/\344\270\255\346\226\207/lib-python/3/encodings", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    stat("/home/yen/tmp/virtualenv/\344\270\255\346\226\207/lib-python/3/encodings/__init__.py", {st_mode=S_IFREG|0644, st_size=5067, ...}) = 0
    stat("/home/yen/tmp/virtualenv/\357\277\275\357\277\275\357\277\275\357\277\275\357\277\275\357\277\275/lib-python/3/encodings/__init__", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    stat("/home/yen/tmp/virtualenv/\357\277\275\357\277\275\357\277\275\357\277\275\357\277\275\357\277\275/lib-python/3/encodings/__init__.py", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    stat("/home/yen/tmp/virtualenv/\357\277\275\357\277\275\357\277\275\357\277\275\357\277\275\357\277\275/lib-python/3/encodings/__init__.pyc", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    stat("/home/yen/tmp/virtualenv/\357\277\275\357\277\275\357\277\275\357\277\275\357\277\275\357\277\275/lib-python/3/encodings/__init__.pypy3-24.so", 0x7f2fe32f3030) = -1 ENOENT (No such file or directory)
    close(4)                                = 0
    write(2, "debug: OperationError:\n", 23debug: OperationError:
    ) = 23
    write(2, "debug:  operror-type: LookupErro"..., 34debug:  operror-type: LookupError
    ) = 34
    write(2, "debug:  operror-value: no codec "..., 81debug:  operror-value: no codec search functions registered: can't find encoding
    ) = 81
    write(2, "debug: OperationError:\n", 23debug: OperationError:
    ) = 23
    write(2, "debug:  operror-type: AttributeE"..., 37debug:  operror-type: AttributeError
    ) = 37
    write(2, "debug:  operror-value: stdout\n", 30debug:  operror-value: stdout
    ) = 30
    exit_group(1)                           = ?
    +++ exited with 1 +++
    

    b'\344\270\255\346\226\207'.decode('utf-8') is exactly 中文, while b'\357\277\275\357\277\275\357\277\275\357\277\275\357\277\275\357\277\275' is simply UTF-8 BOM repeated 6 times.

  2. Chi Hsuan Yen reporter

    Reproducible on hg py3k branch. The following commands lead to UnicodeDecodeError:

    $ mkdir 中文 && cd 中文
    $ hg clone -b py3k https://bitbucket.org/pypy/pypy && cd pypy
    $ python2 pypy/bin/pyinteractive.py
    
  3. Philip Jenvey

    Chi, thank you for this report

    The original issue with the virtualenv should be fixed as of a74ce66 (and the earlier b5da016 is probably required as well FYI)

    I could see it happening when creating the virtualenv or with just plane running a pypy with a PYTHONHOME/executable including such characters in their paths

    The issue with pyinteractive was a slightly different problem that should be solved by an also earlier 2dd24a7

    If you have a chance please try it again (there are some nightly builds here http://buildbot.pypy.org/nightly/py3k/ )

  4. Log in to comment