tox doesn't strip line coments from dependencies

Issue #332 on hold
Florian Bruhin created an issue

With an up-to-date virtualenv (which downloads an up-to-date pip), since pip 8.1.2, tox breaks when there's a tox.ini like this:

[testenv]
deps=
  pytest # comment

output:

GLOB sdist-make: /home/florian/tmp/setup.py
python create: /home/florian/tmp/.tox/python
python installdeps: pytest # comment
ERROR: invocation failed (exit code 1), logfile: /home/florian/tmp/.tox/python/log/python-1.log
ERROR: actionid: python
msg: getenv
cmdargs: [local('/home/florian/tmp/.tox/python/bin/pip'), 'install', 'pytest # comment']
env: {...}

Invalid requirement: 'pytest # comment'
Traceback (most recent call last):
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/packaging/requirements.py", line 92, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 1172, in parseString
    raise exc
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 1162, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 2462, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 1032, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/pyparsing.py", line 2265, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd (at char 7), (line:1, col:8)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/req/req_install.py", line 78, in __init__
    req = Requirement(req)
  File "/home/florian/tmp/.tox/python/lib/python3.4/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'# commen'"


ERROR: could not install deps [pytest # comment]; v = InvocationError('/home/florian/tmp/.tox/python/bin/pip install pytest # comment (see /home/florian/tmp/.tox/python/log/python-1.log)', 1)

This is because pip doesn't handle comments as part of a commandline argument since that version anymore: https://github.com/pypa/pip/issues/3667

They replied in that issue that it isn't really intended that it worked before, and I tend to agree.

Marking as critical as this completely breaks tox unless you set VIRTUALENV_DOWNLOAD=no and never upgrade virtualenv.

I think tox should either strip those comments (no idea how tricky that is as pip URLs can include hashes as well), or write temporary requirement.txt files and direct pip to them.

Comments (8)

  1. Ronny Pfannschmidt

    as far as i understand, pip urls only contain hashes as part of scm metadata, those dont have whitespace right before as such i think we can skip anything after r'[ \t]#'

  2. Florian Bruhin reporter

    FWIW the underlying pip issue has been closed now and the behaviour won't be changed back on their side.

  3. Log in to comment