Don't rely on 2to3 to properly rewrite execfile()

Issue #349 resolved
Barry Warsaw
created an issue

distribute uses execfile('', ...) in setuptools/, and relies on 2to3 to rewrite this to exec(compile(open('').read(), ...) however this breaks Python 3 installations where the has a BOM.

The reason is that Python 3's open('') by default opens the file in text mode, which leaves the BOM as a leading unprintable \ufeff character. This crashes compile() with a SyntaxError.

As an example, try to pip install mimeparse into a Python 3.3 virtualenv. mimeparse's has a BOM.

Python's built-in compile() does the right thing when the BOM'd is read in binary mode though, as you can see by explicitly using compile(open('', 'rb').read(), ...)

Should 2to3 be rewriting execfile() into a binary open? I'm not sure. But it would be easy enough for distribute to explicitly use this pattern (which should be safe in Python 2 as well) instead of relying on 2to3 to rewrite the execfile() call correctly.

Comments (4)

  1. Barry Warsaw reporter

    Just for others running across this issue, the workaround is to not use pip to install into the virtualenv. Download the mimeparse tarball, unpack it, and run python install into the venv. This works because you're not relying on distribute to execfile() the

  2. Erik Bray

    I'd say it's silly that there's a BOM, especially in a utf-8 encoded file. But in any case this is easy enough to fix on distribute's end that it probably should be fixed.

  3. Log in to comment