Upload runs with wrong Python

Issue #395 on hold
Jason R. Coombs
created an issue

Trying to use devpi upload --only-docs the run crashes because it's grabbing an old Python that's not compatible with the environment. The same issue occurs with a plain upload:

$ rwt sphinx -r docs/requirements.txt -- -m devpi upload             
Collecting sphinx
  Using cached Sphinx-1.5.3-py2.py3-none-any.whl
Collecting rst.linker (from -r docs/requirements.txt (line 1))
  Using cached rst.linker-1.8.2-py2.py3-none-any.whl
Collecting Pygments>=2.0 (from sphinx)
  Using cached Pygments-2.2.0-py2.py3-none-any.whl
Collecting snowballstemmer>=1.1 (from sphinx)
  Using cached snowballstemmer-1.2.1-py2.py3-none-any.whl
Collecting requests>=2.0.0 (from sphinx)
  Using cached requests-2.13.0-py2.py3-none-any.whl
Collecting babel!=2.0,>=1.3 (from sphinx)
  Using cached Babel-2.3.4-py2.py3-none-any.whl
Collecting Jinja2>=2.3 (from sphinx)
  Using cached Jinja2-2.9.5-py2.py3-none-any.whl
Collecting six>=1.5 (from sphinx)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting alabaster<0.8,>=0.7 (from sphinx)
  Using cached alabaster-0.7.10-py2.py3-none-any.whl
Collecting docutils>=0.11 (from sphinx)
  Using cached docutils-0.13.1-py3-none-any.whl
Collecting imagesize (from sphinx)
  Using cached imagesize-0.7.1-py2.py3-none-any.whl
Collecting python-dateutil (from rst.linker->-r docs/requirements.txt (line 1))
  Using cached python_dateutil-2.6.0-py2.py3-none-any.whl
Collecting pytz>=0a (from babel!=2.0,>=1.3->sphinx)
  Using cached pytz-2016.10-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.3->sphinx)
  Using cached MarkupSafe-1.0.tar.gz
Installing collected packages: Pygments, snowballstemmer, requests, pytz, babel, MarkupSafe, Jinja2, six, alabaster, docutils, imagesize, sphinx, python-dateutil, rst.linker
  Running setup.py install for MarkupSafe ... done
Successfully installed Jinja2-2.9.5 MarkupSafe-1.0 Pygments-2.2.0 alabaster-0.7.10 babel-2.3.4 docutils-0.13.1 imagesize-0.7.1 python-dateutil-2.6.0 pytz-2016.10 requests-2.13.0 rst.linker-1.8.2 six-1.10.0 snowballstemmer-1.2.1 sphinx-1.5.3
detected devpi:upload section in /Users/jaraco/Dropbox/code/yg/G/queso/setup.cfg
using workdir /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi2
copied repo /Users/jaraco/Dropbox/code/yg/G/queso/.git to /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi2/upload/queso/.git
pre-build: cleaning /Users/jaraco/Dropbox/code/yg/G/queso/dist
-->  /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi2/upload/queso$ /usr/local/bin/python setup.py sdist --formats gztar 
Traceback (most recent call last):
  File "setup.py", line 75, in <module>
    setuptools.setup(**params)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line 51, in run
    self.run_command(cmd_name)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 970, in run_command
    cmd_obj = self.get_command_obj(command)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 845, in get_command_obj
    klass = self.get_command_class(command)
  File "/usr/local/lib/python2.7/site-packages/setuptools/dist.py", line 492, in get_command_class
    return _Distribution.get_command_class(self, command)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 815, in get_command_class
    __import__ (module_name)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/check.py", line 13, in <module>
    from docutils.utils import Reporter
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-igrsglo7/docutils/utils/__init__.py", line 20, in <module>
    import docutils.io
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-igrsglo7/docutils/io.py", line 340
    (self.destination.mode, mode)), file=self._stderr)
                                        ^
SyntaxError: invalid syntax
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/devpi/__main__.py", line 3, in <module>
    main.main()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/devpi/main.py", line 32, in main
    return method(hub, hub.args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/devpi/upload.py", line 41, in main
    default_formats=setupcfg.get("formats")))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/devpi/upload.py", line 401, in setup_build
    out = self.hub.popen_output(cmd, cwd=self.rootpath)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/devpi/main.py", line 235, in popen_output
    return check_output(args, cwd=str(cwd))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/devpi_common/proc.py", line 18, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', 'setup.py', 'sdist', '--formats', 'gztar']' returned non-zero exit status 1.

The program fails because it's invoking the system python (/usr/local/bin/python), which resolves to Python2.7, which isn't compatible with the Python 3 libraries installed for sphinx.

Obviously, there I'm using rwt to resolve the sphinx and doc build dependencies, but you can encounter the same issue with a virtualenv:

$ python -m venv ~/.envs/devpi-doc-upload
$ ~/.envs/devpi-doc-upload/bin/pip install sphinx -r docs/requirements.txt 
Collecting sphinx
  Using cached Sphinx-1.5.3-py2.py3-none-any.whl
Collecting rst.linker (from -r docs/requirements.txt (line 1))
  Using cached rst.linker-1.8.2-py2.py3-none-any.whl
Collecting imagesize (from sphinx)
  Using cached imagesize-0.7.1-py2.py3-none-any.whl
Collecting Jinja2>=2.3 (from sphinx)
  Using cached Jinja2-2.9.5-py2.py3-none-any.whl
Collecting alabaster<0.8,>=0.7 (from sphinx)
  Using cached alabaster-0.7.10-py2.py3-none-any.whl
Collecting requests>=2.0.0 (from sphinx)
  Using cached requests-2.13.0-py2.py3-none-any.whl
Collecting six>=1.5 (from sphinx)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting docutils>=0.11 (from sphinx)
  Using cached docutils-0.13.1-py3-none-any.whl
Collecting babel!=2.0,>=1.3 (from sphinx)
  Using cached Babel-2.3.4-py2.py3-none-any.whl
Collecting Pygments>=2.0 (from sphinx)
  Using cached Pygments-2.2.0-py2.py3-none-any.whl
Collecting snowballstemmer>=1.1 (from sphinx)
  Using cached snowballstemmer-1.2.1-py2.py3-none-any.whl
Collecting python-dateutil (from rst.linker->-r docs/requirements.txt (line 1))
  Using cached python_dateutil-2.6.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.3->sphinx)
  Using cached MarkupSafe-1.0.tar.gz
Collecting pytz>=0a (from babel!=2.0,>=1.3->sphinx)
  Using cached pytz-2016.10-py2.py3-none-any.whl
Installing collected packages: imagesize, MarkupSafe, Jinja2, alabaster, requests, six, docutils, pytz, babel, Pygments, snowballstemmer, sphinx, python-dateutil, rst.linker
  Running setup.py install for MarkupSafe ... done
Successfully installed Jinja2-2.9.5 MarkupSafe-1.0 Pygments-2.2.0 alabaster-0.7.10 babel-2.3.4 docutils-0.13.1 imagesize-0.7.1 python-dateutil-2.6.0 pytz-2016.10 requests-2.13.0 rst.linker-1.8.2 six-1.10.0 snowballstemmer-1.2.1 sphinx-1.5.3
$ ~/.envs/devpi-doc-upload/bin/pip install devpi-client
Collecting devpi-client
  Downloading devpi_client-2.7.0-py2.py3-none-any.whl
Collecting devpi-common<4.0,>2.0.2 (from devpi-client)
  Downloading devpi_common-3.0.1-py2.py3-none-any.whl
Collecting tox>=1.7.1 (from devpi-client)
  Using cached tox-2.6.0-py2.py3-none-any.whl
Collecting pkginfo>=1.2b1 (from devpi-client)
  Downloading pkginfo-1.4.1-py2.py3-none-any.whl
Collecting py>=1.4.31 (from devpi-client)
  Using cached py-1.4.32-py2.py3-none-any.whl
Collecting check-manifest>=0.28 (from devpi-client)
  Cache entry deserialization failed, entry ignored
  Using cached check_manifest-0.35-py2.py3-none-any.whl
Requirement already satisfied: requests>=2.3.0 in /Users/jaraco/.envs/devpi-doc-upload/lib/python3.6/site-packages (from devpi-common<4.0,>2.0.2->devpi-client)
Collecting pluggy<1.0,>=0.3.0 (from tox>=1.7.1->devpi-client)
  Downloading pluggy-0.4.0-py2.py3-none-any.whl
Collecting virtualenv>=1.11.2; python_version != "3.2" (from tox>=1.7.1->devpi-client)
  Downloading virtualenv-15.1.0-py2.py3-none-any.whl (1.8MB)
    100% |████████████████████████████████| 1.8MB 601kB/s 
Installing collected packages: py, devpi-common, pluggy, virtualenv, tox, pkginfo, check-manifest, devpi-client
Successfully installed check-manifest-0.35 devpi-client-2.7.0 devpi-common-3.0.1 pkginfo-1.4.1 pluggy-0.4.0 py-1.4.32 tox-2.6.0 virtualenv-15.1.0



$ ~/.envs/devpi-doc-upload/bin/python -m devpi upload --only-docs
detected devpi:upload section in /Users/jaraco/Dropbox/code/yg/G/queso/setup.cfg
using workdir /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi4
copied repo /Users/jaraco/Dropbox/code/yg/G/queso/.git to /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi4/upload/queso/.git
pre-build: cleaning /Users/jaraco/Dropbox/code/yg/G/queso/dist
-->  /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi4/upload/queso$ /usr/local/bin/python setup.py build_sphinx -E --build-dir /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi4/upload/queso/build 
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'build_sphinx'
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/jaraco/.envs/devpi-doc-upload/lib/python3.6/site-packages/devpi/__main__.py", line 3, in <module>
    main.main()
  File "/Users/jaraco/.envs/devpi-doc-upload/lib/python3.6/site-packages/devpi/main.py", line 32, in main
    return method(hub, hub.args)
  File "/Users/jaraco/.envs/devpi-doc-upload/lib/python3.6/site-packages/devpi/upload.py", line 43, in main
    p = exported.setup_build_docs()
  File "/Users/jaraco/.envs/devpi-doc-upload/lib/python3.6/site-packages/devpi/upload.py", line 417, in setup_build_docs
    "--build-dir", build], cwd=self.rootpath)
  File "/Users/jaraco/.envs/devpi-doc-upload/lib/python3.6/site-packages/devpi/main.py", line 219, in popen_output
    return check_output(args, cwd=str(cwd))
  File "/Users/jaraco/.envs/devpi-doc-upload/lib/python3.6/site-packages/devpi_common/proc.py", line 18, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', 'setup.py', 'build_sphinx', '-E', '--build-dir', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi4/upload/queso/build']' returned non-zero exit status 1.

The reason build_sphinx is missing is because the Python being used is the system Python 2.7 which doesn't have Sphinx installed.

I haven't yet looked into the code, but I suspect that these build steps should use sys.executable instead of 'python' for their subprocess operations.

Comments (13)

  1. Jason R. Coombs reporter

    As a workaround, I found that activating the virtualenv would work around the issue (by making python resolve to the virtualenv). I'll test now to see if sys.executable fixes the issue.

  2. Jason R. Coombs reporter

    Yes, using sys.executable corrects for the issue:

    $ hg diff -R ~/p/devpi
    diff -r 789fe1bfb106 client/devpi/upload.py
    --- a/client/devpi/upload.py    Sat Mar 04 13:23:30 2017 +0000
    +++ b/client/devpi/upload.py    Fri Mar 10 14:14:13 2017 -0500
    @@ -329,10 +329,7 @@
             self.rootpath = rootpath
             self.origrepo = origrepo
             self.target_distdir = origrepo.join("dist")
    -        python = py.path.local.sysfind("python")
    -        if not python:
    -            raise ValueError("could not find 'python' executable")
    -        self.python = str(python)
    +        self.python = sys.executable
    
         def __str__(self):
             return "<Exported %s>" % self.rootpath
    $ rwt sphinx ~/p/devpi/client -r docs/requirements.txt -- -m devpi upload --only-docs 
    Processing /Users/jaraco/p/devpi/client
    Collecting sphinx
      Using cached Sphinx-1.5.3-py2.py3-none-any.whl
    Collecting rst.linker (from -r docs/requirements.txt (line 1))
      Using cached rst.linker-1.8.2-py2.py3-none-any.whl
    Collecting tox>=1.7.1 (from devpi-client==2.7.0)
      Using cached tox-2.6.0-py2.py3-none-any.whl
    Collecting devpi_common<4.0,>2.0.2 (from devpi-client==2.7.0)
      Using cached devpi_common-3.0.1-py2.py3-none-any.whl
    Collecting pkginfo>=1.2b1 (from devpi-client==2.7.0)
      Using cached pkginfo-1.4.1-py2.py3-none-any.whl
    Collecting check-manifest>=0.28 (from devpi-client==2.7.0)
      Using cached check_manifest-0.35-py2.py3-none-any.whl
    Collecting py>=1.4.31 (from devpi-client==2.7.0)
      Using cached py-1.4.32-py2.py3-none-any.whl
    Collecting alabaster<0.8,>=0.7 (from sphinx)
      Using cached alabaster-0.7.10-py2.py3-none-any.whl
    Collecting imagesize (from sphinx)
      Using cached imagesize-0.7.1-py2.py3-none-any.whl
    Collecting babel!=2.0,>=1.3 (from sphinx)
      Using cached Babel-2.3.4-py2.py3-none-any.whl
    Collecting Pygments>=2.0 (from sphinx)
      Using cached Pygments-2.2.0-py2.py3-none-any.whl
    Collecting snowballstemmer>=1.1 (from sphinx)
      Using cached snowballstemmer-1.2.1-py2.py3-none-any.whl
    Collecting docutils>=0.11 (from sphinx)
      Using cached docutils-0.13.1-py3-none-any.whl
    Collecting requests>=2.0.0 (from sphinx)
      Using cached requests-2.13.0-py2.py3-none-any.whl
    Collecting six>=1.5 (from sphinx)
      Using cached six-1.10.0-py2.py3-none-any.whl
    Collecting Jinja2>=2.3 (from sphinx)
      Using cached Jinja2-2.9.5-py2.py3-none-any.whl
    Collecting python-dateutil (from rst.linker->-r docs/requirements.txt (line 1))
      Using cached python_dateutil-2.6.0-py2.py3-none-any.whl
    Collecting virtualenv>=1.11.2; python_version != "3.2" (from tox>=1.7.1->devpi-client==2.7.0)
      Using cached virtualenv-15.1.0-py2.py3-none-any.whl
    Collecting pluggy<1.0,>=0.3.0 (from tox>=1.7.1->devpi-client==2.7.0)
      Using cached pluggy-0.4.0-py2.py3-none-any.whl
    Collecting pytz>=0a (from babel!=2.0,>=1.3->sphinx)
      Using cached pytz-2016.10-py2.py3-none-any.whl
    Collecting MarkupSafe>=0.23 (from Jinja2>=2.3->sphinx)
      Using cached MarkupSafe-1.0.tar.gz
    Installing collected packages: alabaster, imagesize, pytz, babel, Pygments, snowballstemmer, docutils, requests, six, MarkupSafe, Jinja2, sphinx, python-dateutil, rst.linker, virtualenv, py, pluggy, tox, devpi-common, pkginfo, check-manifest, devpi-client
      Running setup.py install for MarkupSafe ... done
      Running setup.py install for devpi-client ... done
    Successfully installed Jinja2-2.9.5 MarkupSafe-1.0 Pygments-2.2.0 alabaster-0.7.10 babel-2.3.4 check-manifest-0.35 devpi-client-2.7.0 devpi-common-3.0.1 docutils-0.13.1 imagesize-0.7.1 pkginfo-1.4.1 pluggy-0.4.0 py-1.4.32 python-dateutil-2.6.0 pytz-2016.10 requests-2.13.0 rst.linker-1.8.2 six-1.10.0 snowballstemmer-1.2.1 sphinx-1.5.3 tox-2.6.0 virtualenv-15.1.0
    detected devpi:upload section in /Users/jaraco/Dropbox/code/yg/G/queso/setup.cfg
    using workdir /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi6
    copied repo /Users/jaraco/Dropbox/code/yg/G/queso/.git to /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi6/upload/queso/.git
    pre-build: cleaning /Users/jaraco/Dropbox/code/yg/G/queso/dist
    -->  /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi6/upload/queso$ /Library/Frameworks/Python.framework/Versions/3.6/bin/python3 setup.py build_sphinx -E --build-dir /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/devpi6/upload/queso/build 
    ...
    built: /Users/jaraco/Dropbox/code/yg/G/queso/dist/queso-36.7.doc.zip [sphinx docs] 666.899kb
    doc_upload of queso-36.7.doc.zip to https://devpi-master.yougov.net/root/yg/
    
  3. Florian Schulze

    Hmm, after thinking about it, I think sys.executable isn't the correct solution for everyone either. I have devpi globally installed, but always expect that the stuff from my current virtualenv is used, because different packages have different requirements for uploads.

    What about detecting a virtualenv, use it if found and otherwise use sys.executable?

  4. Jason R. Coombs reporter

    Good point. We don't necessarily want to use the environment where devpi client is installed to invoke the builds, although this is what other tools like pip do.

    I presume when you say "current virtualenv", you mean an activated virtualenv.

    Looking at the activate scripts on late releases of both virtualenv and pyvenv, they both export the VIRTUAL_ENV variable, so that might be a suitable signal to detect if Python should be found in the path or if sys.executable should be used.

    Perhaps something like this:

    $ hg -R ~/p/devpi diff
    diff -r 789fe1bfb106 client/devpi/upload.py
    --- a/client/devpi/upload.py    Sat Mar 04 13:23:30 2017 +0000
    +++ b/client/devpi/upload.py    Sat Mar 11 12:08:00 2017 -0500
    @@ -329,10 +329,20 @@
             self.rootpath = rootpath
             self.origrepo = origrepo
             self.target_distdir = origrepo.join("dist")
    +
    +    @property
    +    def python(self):
    +        """
    +        Find the best Python executable for invoking builds and
    +        uploads. Uses a virtualenv if one is activated, but otherwise
    +        falls back to sys.executable, the Python under which devpi
    +        client is running.
    +        """
    +        return self._virtualenv_python() or sys.executable
    +
    +    def _virtualenv_python(self):
             python = py.path.local.sysfind("python")
    -        if not python:
    -            raise ValueError("could not find 'python' executable")
    -        self.python = str(python)
    +        return os.environ.get('VIRTUAL_ENV') and python and str(python)
    
         def __str__(self):
             return "<Exported %s>" % self.rootpath
    
  5. Jason R. Coombs reporter

    I like That, especially if the syntax matches that of virtualenvs option. Especially nice if it resolves absolute and relative paths and even py -3 (pylauncher) invocations where available.

  6. Jason R. Coombs reporter

    I should say though, the inference for a virtualenv seems more elegant in at least the use case Florian has described. I hate to think he'll have to pass an option with essentially every invocation.

  7. Jason R. Coombs reporter

    Yes, that sounds like a good plan.

    What's pylauncher

    pylauncher is a Windows executable that comes bundled/installed with the Python.org installer (Python 3 only). It allows scripts to indicate which Python version they run under (honors shebang lines) but also allows for launching a specific python version with py -3.3 or py -2 or py -3-x32 (Python 3 32-bit). It supplements the usage we find in Unix systems invoking python2 or python3.3.

    It's conceivable the launcher could also be ported to Unix, providing a platform-independent way to invoke specific Python versions, but that's more of a dream than a plan.

  8. Log in to comment