1. Holger Krekel
  2. tox

Issues

Issue #112 new

support setup.py scripts that build multiple packages

Peter Feiner
created an issue

I'd like to be able to use tox with setup.py scripts that build multiple packages (i.e., they make multiple setup() calls). The problem is that tox only installs the sdist zip archive from one package into the venv instead of installing all of them. Here's a minimal example:

peter@gremlin ~/foobar % find . 
.
./tox.ini
./setup.py
peter@gremlin ~/foobar % cat tox.ini
[tox]
envlist=py27

[testenv]
whitelist_externals=true
commands=true
peter@gremlin ~/foobar % cat setup.py 
#!/usr/bin/env python
from distutils.core import setup
setup(name='foo', install_requires=['bar'])
setup(name='bar')
peter@gremlin ~/foobar % tox -v
using tox.ini: /home/peter/foobar/tox.ini
using tox-1.5.0 from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc
GLOB sdist-make: /home/peter/foobar/setup.py
  /home/peter/foobar$ /usr/bin/python /home/peter/foobar/setup.py sdist --formats=zip --dist-dir /home/peter/foobar/.tox/dist >/home/peter/foobar/.tox/log/tox-0.log
py27 create: /home/peter/foobar/.tox/py27
  /home/peter/foobar/.tox$ /usr/bin/python2.7 /usr/local/lib/python2.7/dist-packages/virtualenv.py --setuptools py27 >/home/peter/foobar/.tox/py27/log/py27-0.log
py27 inst: /home/peter/foobar/.tox/dist/foo-0.0.0.zip
  /home/peter/foobar/.tox/py27/log$ /home/peter/foobar/.tox/py27/bin/pip install /home/peter/foobar/.tox/dist/foo-0.0.0.zip >/home/peter/foobar/.tox/py27/log/py27-1.log
ERROR: invocation failed, logfile: /home/peter/foobar/.tox/py27/log/py27-1.log
ERROR: actionid=py27
msg=installpkg
cmdargs=[local('/home/peter/foobar/.tox/py27/bin/pip'), 'install', '/home/peter/foobar/.tox/dist/foo-0.0.0.zip']
env={'PYTHONIOENCODING': 'utf_8', 'SSH_CLIENT': '192.168.1.207 59563 22', 'LOGNAME': 'peter', 'USER': 'peter', 'HOME': '/home/peter', 'PATH': '/home/peter/foobar/.tox/py27/bin:/home/peter/test:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm-256color', 'SHELL': '/bin/zsh', 'SHLVL': '1', 'PWD': '/home/peter/foobar', 'EDITOR': 'vim', 'OS_USERNAME': 'admin', 'OS_TENANT_NAME': 'admin', 'OS_PASSWORD': 'admin', '_': '/usr/local/bin/tox', 'SSH_CONNECTION': '192.168.1.207 59563 96.45.203.162 22', 'SSH_TTY': '/dev/pts/74', 'OLDPWD': '/home/peter', 'OS_AUTH_URL': 'http://192.168.16.3:5000/v2.0/', 'MAIL': '/var/mail/peter', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:', 'PAGER': 'less'}
Unpacking /home/peter/foobar/.tox/dist/foo-0.0.0.zip
  Running setup.py egg_info for package from file:///home/peter/foobar/.tox/dist/foo-0.0.0.zip


Downloading/unpacking bar (from foo==0.0.0)
  Could not find any downloads that satisfy the requirement bar (from foo==0.0.0)
Cleaning up...
No distributions at all found for bar (from foo==0.0.0)
Storing complete log in /home/peter/.pip/pip.log

_______________________________________________________________________________________________________________________________________ summary _______________________________________________________________________________________________________________________________________
ERROR:   py27: InvocationError: /home/peter/foobar/.tox/py27/bin/pip install /home/peter/foobar/.tox/dist/foo-0.0.0.zip (see /home/peter/foobar/.tox/py27/log/py27-1.log)

Here's a hacky patch to get the job done:

diff -r 3be5bca16f8b tox/_venv.py
--- a/tox/_venv.py  Thu Aug 08 13:27:01 2013 +0200
+++ b/tox/_venv.py  Thu Aug 08 14:48:38 2013 -0400
@@ -245,7 +245,9 @@
         else:
             action.setactivity("inst-nodeps", sdistpath)
             extraopts = ['-U', '--no-deps']
-        self._install([sdistpath], extraopts=extraopts, action=action)
+        import glob
+        sdistpaths = glob.glob('%s/*.zip' % sdistpath.dirname)
+        self._install(sdistpaths, extraopts=extraopts, action=action)

     def install_deps(self, action=None):
         if action is None:

I could prepare a proper patch if there's interest.

Comments (5)

  1. Peter Feiner reporter

    Although a single setup() call is typical, I don't think it's a hard requirement. The output of --fullname is

    foo-0.0.0
    bar-0.0.0
    

    Each call to setup() is effectively like running a different setup.py script with its own setup function. So as long as two setup.py scripts can coexist in a directory, then two setup() calls within the same setup.py should be OK. Some practical considerations are how you'll deal with some implicit inputs to the various setup() calls being the same; for example, MANIFEST.in has to work with all setup()s -- or it can be excluded entirely.

    Of course, if a tool like tox calls setup.py --fullname and expects one line of ouptut, then it's not going to work. But tox doesn't seem to use setup.py --fullname :-). Infact, the only problem I had with tox was the assumption that one sdist .zip file was created.

  2. Holger Krekel repo owner

    Maybe it's a workable idea, your patch looks easy enough. If we went down that road, we also need to add support for multiple "--installpkg" options. So please prepare a patch/PR with tests for this if you'd like to see this happen. I am going to review only early September though -- gone to vacation from tomorrow.

  3. Raphaël Hertzog

    Peter, you never replied to Holger. It would be nice to get this feature merged. I use something similar but everything just magically works because I use sitepackages=True and my modules are found outside of the virtualenv (via the current directory).

  4. Log in to comment