pkg_resources.py... ScriptMaker...?

C Anthony Risinger avatarC Anthony Risinger created an issue

this is more of a question... direct me elsewhere if more appropriate.

ENVIRON

setuptools (0.9.5)

wheel (0.18.0)

distlib (HEAD)

my setup generates wheels for everything (including setuptools/wheel/distlib), and installs with wheel module (this will be moved to distlib as time allows), thus i have 100% non-legacy environment.

the problem is the pkg_resources in current setuptools completely chokes when distributions are installed to a non-root zipfile, eg. path/to/python.zip/lib/python2.7/site-packages/*.dist-info. granted this is a setuptools problem i need to report (and furthermore, unzipped eggs within such a zipfile will cause exponential scanning of itself for each egg added, and never completes)... is there a nice way to handle this in distlib alone? already? other thoughts?

i saw the pkg_resources.py impl submitted to pip (https://github.com/pypa/pip/pull/909):

https://raw.github.com/vsajip/pip/e0025790cce57d4fcfaba8737a3aa512cf0e1981/pip/compat/pkg_resources.py

...and tried that out (after fix _legacy_key), but it is missing load_entry_point :(

ATM, load_entry_point is the only thing i need and it seems easy-ish to implement, but i thought i'd drop a note here to learn more about the problem itself, or any other relevant infos... thoughts?

Comments (7)

  1. Vinay Sajip

    In terms of what you need to do, what is missing from distlib which makes you use the wheel package? For custom builds for which you need setuptools, see my wheeler.py script which builds wheels using vanilla (not needing wheel-enabled) pip.

    In terms of load_entry_point, there's a documented equivalent in distlib :-)

    If either of those don't meet your needs, tell me why and I'll take a look.

    One more thing - you mentioned ScriptMaker in the title but not the body. The script generation in distlib, unlike the equivalent in setuptools, does not require any external software to run (whereas setuptools ties your scripts to pkg_resources).

    There are some changes afoot to how script generation works with wheels - see distutils-sig for the latest discussions. Basically, the ideal case is for wheels to not contain any generated scripts or executable launchers when building the wheel, but for the generation and launcher addition happen at the time of installing the wheel.

    Currently, distlib's companion tool distil does it like setuptools - it adds everything at build time. This will change in the future, once the details are specified in PEP 426.

    BTW have you tried out distil as an alternative to pip? It showcases distlib features reasonably well IMO.

  2. C Anthony Risinger

    i don't need to use the wheel package, it's just one of the remaining pieces i haven't yet ported to distlib. i probably don't need setuptools at this point, since i am doing dep resolution manually via distlib, i am really only using it for bdist_wheel -- i don't recall reading in the docs how to actually create a wheel from an sdist? i will look again though as i apparently also skipped over the bits about pkg_resources... whoopsie...

    all i'm really trying to determine is what the best course of action actually is, probably from your perspective, ie: - should i be using distlib to do everything, including the call to setup.py (even though several dists pretty much assume setuptools)? - should i be converting pkg_resources based entry points to distlib ones, after the fact? - ^^^ if so, is something like this already implemented? - should i be intercepting setuptools script creation, and outputting distlib ones instead? - ...what would you do ;) ?

    BTW, i'm not using pip at all (TBH, i don't like it very much and the code is rough); i'm doing something like this:

    # cat <<EOF > setuptools/command/bdist_wheel.py
    from wheel.bdist_wheel import *
    EOF
    

    ...which makes bdist_wheel available immediately; i also ensure setuptools is imported all the time (which is pretty much all pip does).

    i guess i'm a little confused about what is/will be needed... i did try distil too, it was very close to what i wanted, but IIRC, there were a few sore spots that made me decide not to use it... though i cannot recall what, nor am i sure the reasons are still even relevant :/ i do recall one of the larger reasons being that i cannot use distlib to fetch distil, since the source is not generally available in the same way as other dists, which means i would have needed to special case it... not the end of the world, i know, but i didnt want to do that ;)

    tl;dr i want to know the best way to create/use a wheel (and it's entry points!) from an sdist, ideally with only distlib, but if not, i want to know the proper order-of-operations, or the "which tool should do what" bits...

    ...does any of the above rambling make sense?

  3. Daniel Holth

    FYI when wheel is installed "python setup.py bdist_wheel" is available in setuptools-using setup.py without having to modify setuptools. It's just a plugin discovered via entry points.

  4. Vinay Sajip

    To create a wheel from an sdist, I use e.g. (for the nose distribution)

    distil download -d /tmp nose
    distil package -d /tmp --format=wheel /tmp/nose-1.3.0
    

    However, that will only work for distributions which don't do custom stuff in setup.py, such as moving files around, creating new files or changing existing ones before running the actual setup() code. To cater for that (oh, the evils of executable setup.py), you would have to use the distil pip command to use setuptools/pip to build wheels, but then you have to have pip available on the path.

    As to whether you should convert pkg_resources entry points - that depends on detailed considerations where you would know better than I do whether it would work for you. If you try it and run into any problems because of how I've done things, I'd be receptive to make changes, as the point is to be functionally equivalent without slavishly copying the exact API.

    I would be interested to know what your sticking points with distil were, especially as you say it was very close to your needs. Feedback is what makes things better :-)

    You can't use distlib to fetch distil, because that is back-to-front: distlib is only a library, whereas distil is an application. The reason it is packaged the way it is is purely as a convenience - I can hardly advocate using pip to install if it's supposed to work without pip, and even supplant pip for many use cases. The big advantage of distil is that you just download it once and copy it to a folder on your path, make it executable and then it works on user site-packages (default), system site-packages (--system) or in any virtual environment (-e venv). While the source is not visible in a repo for browsing, it's all there in the distil script and you could (relatively) easily unzip it to look at if you really want. But, it's currently experimental and changes quite a bit, and I doubt if anybody has the time to really sit down and review it. Of course there is a trust issue as well, but that applies equally to anything installed from PyPI, as setup.py from a random distribution could do just about anything :-)

    If you do try distil to build wheels and run into problems, please let me know and I'll try to be responsive about resolving them where possible.

  5. C Anthony Risinger

    Daniel Holth yeah i did see that once installed it works, the issue was that wheel/setuptools are early requirements for my tool, but i don't actually treat them any different than other dists... ie. they are a normal requirement found in one of [potentially] several processed requirement files. the perceived advantage was being able to bdist_wheel a vanilla setuptools with itself, before anything was installed.

    my setup is a little "magic-y" so i apologize if i'm not conveying enough info... the tool builds python, then uses that python to build all the requirements, before installing python itself... this is because i have code that passively monitors the creation of .so libraries, and generates corresponding Setup + <extension>.[ao] files suitable for static linking. so after all requirements are built, python core is relinked with all the additional objects, resulting in a 100% builtin binary.

    vinay.sajip i am kind of using distlib in the manner you mentioned for distil; i use a prebuilt [packed! directly on sys.path, like an egg] wheel throughout the process to guarantee that distlib is always available, and so far that has worked fine; i'm not too worried about trust-issues/etc (for pretty much the reasons you put forward), i simply want to nail down as solid an implementation as possible in the coming weeks so i can get this beast in the wild :)

    i'm so far quite liking my current integration with distlib directly (which yesterday i finished enough to achieve a successful build, save the pkg_resources probs) but i will revisit distil and gather any shortcomings i experience, and will very likely look to the source for other inspiration... i'm pretty hell-bent on never downloading or needing pip :)

    sort of a side comment, but i read stuff on distutils-sig regarding the perils of using zipped eggs, which is a great loss in my mind, for numerous reasons i'd elaborate if prodded... i know whl is "just an installation format", but i very very very much hope there is a solid (and widely accepted!) resources/loader/[u]mount API, as distlib would hint...

    ...dists that are needlessly not zip-safe are the only thing preventing me from achieving:

    any python application in => one fully self-contained file out

    ...i also have meta_path importers that can pull from the network with working get_data impls... let's make it happen!

    anyways, i'll be back when i have more useful/concrete/coherent information.

  6. Vinay Sajip

    Re. the mountable wheels API - I'm told that that approach with other tools has caused a lot of problems in the past for some people, but I don't have the specifics. So I've left the API in for now, it can always be pulled later (before 1.0) if it's considered too dangerous, and there's some hard evidence of failure modes. For pure-Python wheels you can just add the wheel to sys.path (mount() does that for you) - it's only for wheels with C extensions where you actually need to mount() to be able to access the extensions).

    As with other distlib functionality - feel free to try out wheel mounting where you think you need it, report any problems you encounter and I'll try to make them go away.

    Also remember - there may be some changes soon to the wheel PEP relating to handling of generated scripts and native launchers.

  7. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.