1. Holger Krekel
  2. tox

Issues

Issue #66 closed

Tox shouldn't call pip directly to avoid shebang limitations

Gustavo Picon
created an issue

When the python interpreter for an environment is in a deep directory hierarchy (in my case due to a jenkins matrix with 4 axis), tox will make a pip call like:

../bin/pip

The shebang in this file has the full path for the python interpreter in the virtualenv. In one of my failing tests it was:

#!/home/tabo/jenkins/workspace/tbtest3/TOX_DB/postgres/TOX_DJANGO/dj14/TOX_PYTHON/py27/os/linux/.tox/py27-dj14-postgres/bin/python2.7

With a path length of 131 that exceeds linux's limitation of 127. Patching tox to call $PYTHON_BIN_PATH $PIP_BIN_PATH instead of just $PIP_BIN_PATH solves the problem.

Comments (9)

  1. Glen Nelson

    This is still an issue I hit (running tox 1.7.2). Has this been fixed in any newer versions and missed being closed, or is it still an open defect?

    Thanks!

  2. Amber Brown

    Yeah, I'm hitting this too on Twisted, as we try and move tox-first, including on the Windows builders.

    With the install_command workaround, it works better, but then fails right after with:

    GLOB sdist-make: /Users/red/code/tw spc/setup.py
    py27-tests create: /Users/red/code/tw spc/build/py27-tests
    py27-tests installdeps: cryptography >= 0.9, pyopenssl >= 0.13, service_identity, idna >= 0.6, pyserial, python-subunit, pycrypto, appdirs, soappy
    py27-tests inst: /Users/red/code/tw spc/build/dist/Twisted-16.1.1.zip
    Traceback (most recent call last):
      File "/Users/red/venvs/commands/bin/tox", line 11, in <module>
        sys.exit(cmdline())
      File "/Users/red/venvs/commands/lib/python2.7/site-packages/tox/session.py", line 39, in main
        retcode = Session(config).runcommand()
      File "/Users/red/venvs/commands/lib/python2.7/site-packages/tox/session.py", line 373, in runcommand
        return self.subcommand_test()
      File "/Users/red/venvs/commands/lib/python2.7/site-packages/tox/session.py", line 540, in subcommand_test
        output = py.process.cmdexec("%s freeze" % (pip))
      File "/Users/red/venvs/commands/lib/python2.7/site-packages/py/_process/cmdexec.py", line 28, in cmdexec
        raise ExecutionFailed(status, status, cmd, out, err)
    py.process.cmdexec.Error: ExecutionFailed: 127  /Users/red/code/tw spc/build/py27-tests/bin/pip freeze
    /bin/sh: /Users/red/code/tw: No such file or directory
    

    I believe https://bitbucket.org/hpk42/tox/pull-requests/172/do-not-call-pip-directly-trying-to-fix/diff should fix this issue.

  3. Holger Krekel repo owner

    So it turns out defaulting to "python -m pip" is a breaking change we cannot do. The problem is that this invocation adds the current working directory and this makes pip think a package is installed if it is in the invocation current working directory. You can change "install_command" to use "python -m pip" if that works for you (e.g. if you don't have packages in your root dir) or, probably a better idea, use the new "--workdir" option to have a tox working directory which is not deeply nested. The latter method has the added benefit that also all other invocations of scripts in a virtualenv will work. Even if we made "python -m pip" work you would still not be able to invoke other installed scripts in venvs that have too long path names.

  4. Holger Krekel repo owner

    addresses issue66: add --workdir option to override where tox stores its ".tox" directory to "pip freeze" to obtain the list of installed packages. Thanks Ted Shaw, Holger Krekel. close issue66: add documentation to jenkins page on how to avoid "too long shebang" lines when calling pip from tox. Note that we can not use "python -m pip install X" by default because the latter adds the CWD and pip will think X is installed if it is there. "pip install X" does not do that.

    (experimental) New feature: When a search for a config file fails, tox tries loading

    → <<cset 11c13b4313a4>>

  5. Holger Krekel repo owner

    addresses issue66: add --workdir option to override where tox stores its ".tox" directory to "pip freeze" to obtain the list of installed packages. Thanks Ted Shaw, Holger Krekel. close issue66: add documentation to jenkins page on how to avoid "too long shebang" lines when calling pip from tox. Note that we can not use "python -m pip install X" by default because the latter adds the CWD and pip will think X is installed if it is there. "pip install X" does not do that.

    (experimental) New feature: When a search for a config file fails, tox tries loading

    → <<cset 11c13b4313a4>>

  6. Log in to comment