1. PyPA
  2. Python Packaging Authority Projects
  3. wheel
  4. Issues
Issue #92 new

bdist_wheel makes absolute data_files relative to site-packages

Marcus Smith
created an issue

bdist_wheel doesn't handle absolute paths in the "data_files" keyword like standard setuptools installs do, which honor it as absolute (which seems to match the examples in the distutils docs)

when using absolute paths, the data ends up in the packaged wheel at the top level, and get's installed relative to site-packages (along with the project packages)

so, bdist_wheel is re-interpreting distutil's "data_files" differently. maybe better for wheel to fail to build projects with absolute data_files, than to just reinterpret it.

Comments (33)

  1. Marcus Smith reporter

    I.e. it's either that the wheel spec has to grow to cover absolute data_files (I don't see how it could handle them now; putting them into {distribution}-{version}.data doesn't help because that's relative to sys.prefix), or bdist_wheel just needs to fail to build in that case.

  2. Daniel Holth

    Absolute paths need to be allowed but it may be acceptable to restrict to absolute paths within the sdist.

    There's a place in setuptools where certain kinds of paths cause errors and I run into it from time to time. I don't remember the details atm, only that it would be much easier to use if it did allow absolute paths.

  3. Marcus Smith reporter

    Why does it have to be allowed? If bdist_wheel and sdist were consistent, that would be one thing, but they're not and can't be at the current time, so it seems wrong for wheels to build absolute paths and then place them into site-packages

  4. Donald Stufft

    I don't see any reason why absolute paths have to be allowed. I think they are a bad design in general, everything should be rooted in sys.prefix. It's not a very good thing for a Wheel to be able to override /etc/hosts for instance.

  5. Marcus Smith reporter

    btw, there's a metadata issue open for whether wheel would grow the ability to handle platform-specific paths (including absolute I guess) https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdirectory-to-allow-wheels-to

    for me, this issue isn't about that discussion.

    it's about the oddity of placing absolute paths into site-packages

    since wheel has no ability to properly place absolute files currently, it shouldn't build projects that declare them

  6. Marcus Smith reporter

    so take this setup.py which defines an absolute data files at "/opt/data_file": https://gist.github.com/qwcode/9144129 (and assuming there is a "data_file" relative to it)

    build an sdist and wheel and then install each, and see where "data_file" goes.

    • for the sdist: /opt/data_file
    • for the wheel: ../site-packages/opt/data_file

    on the other hand, relative data files get packaged into *.data/data and get installed relative to sys.prefix

  7. Michael Hoglan

    Graphite does a similar thing, not specifically their data files, but the lib files are specified in an absolute location (/opt/graphite/webapp) in the setup.cfg, and it results in the files being under site-packages/opt/graphite/... when you build a wheel and install it in a virtualenv.

    When building from source, I would specify --install-options to change those locations to be relative to the virtualenv, but that does not seem possible to pass those options into pip wheel.

    Removing the prefix / lib configurations in the setup.cfg cause the wheel and source installs to behave the same (ends up in site-packages); Altering the wheel and getting rid of the /opt/graphite/webapp at the top level achieves the same thing (since it would have assumed prefix of . at install);

    btw, I would say its not good practice for a module to be specifying absolute paths... not virtualenv friendly, and I would hope projects would fix that. I see this as more of having to work with projects that are not defined cleanly. And probably allowing there to be consistency between a src install and a wheel install.

  8. Benjamin Bach

    btw, I would say its not good practice for a module to be specifying absolute paths... not virtualenv friendly, and I would hope projects would fix that.


    I see this as more of having to work with projects that are not defined cleanly.

    Well, actually the current problem is to work with package installers and virtualenvs that are defined cleanly!

    Problem is that you may be able to put a data file somewhere using setup(data_files=xx) -- but can you determine where it went from your application instance!?

    That's the main problem I'm facing with setuptools right now... when using setuptools, all paths for the data_files kwarg are relative to sys.prefix, but when installing in a virtualenv, they're not..

  9. pombredanne NA

    Keerthan Jaic the simplest way for me is to only use package data effectively stored in a package directory side by side with the python code that needs them and never use data files. Once you have this, dirname and __file__ will let you navigate to these data file locations relative to your python code location. Since the data is always in the same place relative to the calling code, the fact you are installed globally in a venv or else does not matter anymore.

    As a simple example of this approach:

  10. Benjamin Bach

    My work-around for pip 7.0 (because pip automatically creates wheels from sdists) is to include this in setup.py:

    if 'bdist_wheel' in sys.argv:
        raise RuntimeError("This setup.py does not support wheels")

    Pip will automatically skip the .whl packaging and run the normal sdist installation.

    Why on earth this decision to make an unfinished packaging system deploy things that weren't intended for it by default is beyond my belief :( People who've made sdist installations, released them, and tested them, can create their .whl files themselves... this new bdist_wheel call prolonges the installation process and creates new unexpected behavior.

  11. joe_code

    This is a real problem for me as well.

    My setup.py script works as expected with regard to data_files that use an absolute path and honors them when I do 'python setup.py install' however when I do 'python setup.py bdist_wheel' and then pip install my wheel the data_files that I specified with an absolute path and were correctly installed using a straight setup.py install ARE NOT installed correctly from the wheel and wind up relative to site-packages. I.e. site-packages/usr/lib/blah/blah

    If I want to install a file outside of site-packages (say to an arbitrary place on the filesystem) I should be able to do that. The behaviour is inconsistent. I'd really like to see this fixed because right now I can't use wheels and that's exactly what I want to use.

  12. Benjamin Bach

    joe_code - I can recommend finding a workaround, not using setup.py's setup(). Ultimately, that's what we did, and to be honest and despite my previous harsh rhetoric in this thread, it's nice to get rid of data_files and have a Python project that works inside virtual environments again and can be distributed with Wheel :)

  13. Benjamin Bach

    In our case, we could factor out most of the files in /usr/share and turn them into "package data". The remaining files are now handled by OS installers (for instance debian packages, pkg for Mac, setup.exe for stuff Windows etc).

    In case you don't want to create OS installers, you can have a "run first" approach for your application for which you do if not os.path.exists, possibly adding a file with your project's version in. The disadvantage is uninstallation.

  14. Erik Bray

    I think it's a real problem that this decision has effectively broken a use case that many packages have relied in--in some cases for bad reasons, but in other cases for good reasons.

    Although I personally feel like the reasoning behind the breakage has some merit, breaking things without offering some kind of guidance on how best to handle outside-Python resource files has created yet another sore point against Python packaging that has been raised by some of colleagues, and it's a valid complaint.

    I think the argument "well we shouldn't just allow installing files to arbitrary system locations" is well meaning but ultimately spurious. It's true that, depending on what install_data gets set to, the paths which can be installed to is somewhat limited making it hard, say, to overwrite /etc/hosts. Yet pip will also happily overwrite executables in /usr/bin, for example, which I think is awful and it shouldn't. So really you're making a security-related argument that falls apart because there's actually no promise of security when installing a wheel system-wide (outside a virtualenv). Meanwhile it's possible to hand-craft wheels with files in the .data directory that can be installed almost anywhere within /usr at the very least.

    I think a better approach would be to not make arbitrary decisions for software developers who know what they're doing, and where necessary protect users (and developers who don't know what they're doing) by not allowing pip to overwrite files that already exist on their system (especially for "data files").

  15. Erik Bray

    So treat absolute paths as relative to the root of a virtualenv when installing in a virtualenv, and don't break their semantics on system installs.

  16. Benno Fünfstück

    Is there a way right now to install some file into site-packages that works with both setuptools and bdist_wheel? For example, if I want to install a native library that is later loaded by my application. Or should I not use site-packages for that?

  17. gerard56

    pombredanne NA:

    this seems a bit complex, also https://docs.python.org/2/distutils/setupscript.html?highlight=package_data states that for package data files: 'the files are expected to be part of the package in the source directories' so using a barely documented parameter to work around that seems a not very good thing. (I can't find a mention of using include_package_data on said page of python.org; however there is a threatening warning in python code that using this parameter badly can lead to an infinite recursion, see setuptools/command/sdist.py ...). As there is a simpler way (see below) why do it like you advise ?

    As of the original problem, the said documentation states for the data_files:

    'If directory is a relative path, it is interpreted relative to the installation prefix' Nothing is said explicitly about what the software should do if the directory is an absolute path...

    In fact what it is doing is that really the destination path of data_files is always relative at least when using pip wheel.

    Not very intuitively, when a relative path is specified, the end result is installation higher in the file hierarchy

    '/mypackage', 'myfile.txt' -> installation under <site/dist package>/mypackage

    '/', 'mygoofylibrary.dll' -> installation under <site/dist package> (site if user, dist if admin)

    If I understand correctly, this is what Benno Fünfstück was asking for.

    'etc/myconfig', 'exemple-config.txt' -> installation under /opt/local/etc/myconfig (or /usr/local/etc/myconfig or something like that...) if run as admin .local/etc/myconfig if run as user

    Here the python documentation is very misleading since the given example:

    setup(..., data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']), ('config', ['cfg/data.cfg']), ('/etc/init.d', ['init-script'])] ) will never work, nothing will go in the system /etc/init.d file. as as been said here, the file init-script will end up under <dist packages>/etc/init.d

    And this is indeed a very good thing. If the sysadmin wants to put something in the /etc/init.d directory, the sysadmin should decide it, not having the decision taken by a python non-system package installed blindly. In most cases the /etc directory should stay managed by the computer packaging system.

  18. Log in to comment