Wheel doesn't support environment markers from requirements.txt

Issue #172 new
vitaly numenta
created an issue

Both pip (8.1.2) and setuptools (since 20.6.6) support a common format for environment markers, but wheel ignores the environment markers from requirements.txt.

Environment: OS X El Capitan, pip==8.2.1, setuptools==25.2.0, wheel=0.29.0, python 2.7.10.

I put together a tiny package to demonstrate this here: https://github.com/vitaly-krugl/wheel_marker_fail.

requirements.txt contains a single line: pycapnp==0.5.8 ; platform_system=='BestKeptSecret'

Pip correctly ignores pycapnp because platform_system doesn't match:

$ pip install -r requirements.txt --user
Ignoring pycapnp: markers u"platform_system=='BestKeptSecret'" don't match your environment

In the demonstration package, setup.py reads the lines from requirements.txt and passes them to setuptools.setup() as the install_requires arg. When I execute python setup.py install, setuptools correctly ignores pycapnp, too.

Make an egg via python setup.py bdist_egg and install via easy_install --user dist/wheel_marker_fail-0.0.1-py2.7.egg also honors the markers.

But, unlike all of the other tools, python setup.py bdist_wheel followed by pip install dist/wheel_marker_fail-0.0.1-py2-none-any.whl --user ignores the environment makers and installs pycapnp anyway:

$ pip install dist/wheel_marker_fail-0.0.1-py2-none-any.whl --user
Processing ./dist/wheel_marker_fail-0.0.1-py2-none-any.whl
Collecting pycapnp==0.5.8 (from wheel-marker-fail==0.0.1)
Installing collected packages: pycapnp, wheel-marker-fail
Successfully installed pycapnp-0.5.8 wheel-marker-fail-0.0.1

Comments (17)

  1. vitaly numenta reporter

    Dear @Daniel Holth, thank you for following up with your recommendations.

    A great many packages prefer to keep their python dependencies in a single file - requirements.txt. Numenta's packages are no different. Requirements.txt is an easily-readable single source of truth for dependencies.

    I considered adding extra logic to the production package's setup.py to parse its requirements.txt, identify the lines that contain environment markers, extract the environment markers, and then pass those entries to setup() via the extras_require arg, while passing the rest via the install_requires arg.

    However, it occurred to me that it doesn't make sense for everyone to be complicating the installation code of their packages with such logic. Both pip and setuptools have already made the leap to support this syntax uniformly, and it works beautifully without additional boilerplate. Therefore, it only makes sense for wheel to follow in their footsteps.

    I will take a look at the referenced source code. @Daniel Holth, are you recommending that I try to submit a patch to the wheel repo to support this feature or that I try patching it in my own package's installation logic?

  2. Michael Merickel

    @vitaly numenta I think he is suggesting that the environment markers are supported via adding them to extras_require instead of install_requires. Similar to what wheel is doing here: https://bitbucket.org/pypa/wheel/src/cf4e2d98ecb1f168c50a6de496959b4a10c6b122/setup.py?fileviewer=file-view-default#setup.py-41

    Whether this is the standard way to do it, I have no idea. I agree it's surprising that it works for everything except wheel when you put it in install_requires.

  3. Daniel Holth

    I think it's shocking that people expect to read requires.txt, which can also sometimes contain pip command line arguments, directly into install_requires = []. It is a file format that came up from nowhere but a lot of people seem to like it. Maybe they do not know about 'pip install -e .' or maybe their project doesn't even have setup.py . The way I use requires.txt it would contain the output of 'pip freeze' but install_requires = [] would contain the abstract dependencies (just the names of the direct dependencies).

    Wheel has to know how to translate requirements from install_requires to the dist-info format. It is a different code path than what setuptools uses, and the markers syntax for requires.txt is newer than bdist_wheel.

    A pull request to update bdist_wheel would be fine.

  4. vitaly numenta reporter

    Thanks everyone for the feedback. I am not sufficiently well-versed in python package best practices and have to consider that it's entirely possible that our projects are using a technique that, while convenient for our own internal work-flow, may be non-standard. I will consult with my colleagues.

  5. vitaly numenta reporter

    Our internal work-flow is to run python setup.py install (or python setup.py develop) in our projects and expect that this automatically pre-installs the requirements, hence the copying from requirements.txt into the setup() arg install_requires.

  6. Daniel Holth

    As for extras_require, the syntax is like so

      extras_require={  ':python_version=="2.6"': ['argparse'],
      'signatures': ['keyring', 'keyrings.alt'],
      'signatures:sys_platform!="win32"': ['pyxdg'],  },
    

    That's an empty extra name followed by a colon : and the marker. install_requires is equivalent to an extra with the empty string as its name: extras_require = { '' : [ "install", "requires" ] }

  7. vitaly numenta reporter

    @Daniel Holth, what is the significance of pip install -e . in this thread? pip install --help doesn't relate the -e option to dependencies in any way, so I am not sure what you implied.

    -e, --editable <path/url>   Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.
    

    Thank you.

  8. vitaly numenta reporter

    I see; in your workflow, your expectation is that pip install -e . should install just the abstract dependencies instead of the versioned dependencies from pip freeze. Thank you.

  9. Michael Merickel

    If you have a project that versions some of its dependencies then your workflow should be pip install -r requirements.txt and then pip install -e . (any unpinned deps will be installed from pypi, but anything pinned via requirements.txt will already be there and the pip install -e . will happily keep the pinned version installed).

    Optionally you could bundle the -e . into your requirements.txt and name it something like requirements-dev.txt. This workflow is basically equivalent to your python setup.py develop above except you run pip install -r requirements-dev.txt.

  10. vitaly numenta reporter

    Gentlemen, I have a follow-up question about building the wheel that references the concrete (pinned down dependencies). So, as recommended, I am changing install_requires to contain the abstract dependencies for developers, while having the concrete versioned dependencies in requirements.txt that we use to test official releases of our package.

    What's not clear to me now is how I can possibly build a wheel for deployment to PyPi that records the concrete dependencies, such that when users install my wheel from PyPi, the concrete dependencies would be installed (not the loosely-versioned ones). Users that install my wheel from PyPi will not (and should not need to) have access to my package's requirements.txt, after all. E.g.,

    in setup.py:

    setup(
        extras_require = {
            ":platform_system=='Linux' or platform_system=='Darwin'": ["pycapnp"]
        },
        install_requires=[
          "psutil>=1.0.1,<2"
        ],
        ... <other setup args>
    )
    

    and requirements.txt:

    pycapnp==0.5.8 ; platform_system=='Linux' or platform_system=='Darwin'
    psutil==1.0.1
    

    However, when I build my wheel via python setup.py bdist_wheel, the resulting wheel will reference only the abstract loosely-versioned dependencies from setup.py. Thus, when users install my wheel from PyPi, the wrong versions of dependencies would be installed instead of the concrete ones that I have in requirements.txt.

    I want developers to be able to use the abstract dependencies for experimenting with my package, but installs of my wheel from PyPi need to reference the concrete (pinned-down) dependencies with which my build system actually tested my package.

    What am I missing?

    Many thanks,

    Vitaly

    CC @Daniel Holth @Michael Merickel

    P.S., I posted this question on the Wheel-builders list: https://mail.python.org/pipermail/wheel-builders/2016-September/000220.html

  11. Log in to comment