Possible to have launcher exe find python.exe relative to itself?

[Robert Iannucci Jr.]

(Repost from https://bitbucket.org/pypa/distlib/issues/117)

The scenario I'm facing is that I'd like to be able to distribute python in a 'relocatable' form (think: install to a thumbdrive).

The python interpreter itself seems pretty happy with this arrangement, but the .exe's generated in Scripts (IIUC these are a shim script cated onto the end of one of the launcher .exes in distlib) seem to only support an absolute path to the python interpreter, or a bare exe name (like "python.exe"). The absolute path won't work well for a relocatable distribution for obvious reasons :). Setting it to "python.exe" works slightly better... but it requires that python.exe is on %PATH%, which is not something I can guarantee easily.

Furthermore, I have an ollld python distribution hacked up by one of my teammates (some version of 2.7.6) still using the non-appended archives (i.e. foo.exe launches foo-script.py), and it appears that those scripts begin with #!python.exe and the .exe can find the correct interpreter (likely relative to the .exe launcher)!

It seems that some version of the launcher used to be able to find python.exe relative to itself, but I can't find evidence of this feature in any of the available sources (i.e. anything added to distlib after launcher.c was available).

So, is it possible that this feature: 1) did actually exist in a previous version of the launcher 2) could come back somehow? (I'd be willing to submit a patch if need be :)). Maybe something like "%cd%..\python.exe" so that this behavior doesn't happen magically for "python.exe"?

[Vinay Sajip]

This might actually be working already, without me doing anything special. Just to test the current behaviour, I made a change to the test script make_test.py which combines the command-line executable with a short inline script. I then invoked this with a relative shebang, and it appears to work as expected. See the screenshot:

So, you might want to try this and report back any specific problem you find.

[Thomas Kluyver]

I haven't tried this, but are we confident it's working relative to the folder containing the exe rather than the cwd? I would assume that unless you write specific code for this, relative paths are interpreted from the cwd.

[Vinay Sajip]

I had tried from another folder,

which seemed to be OK, but actually I don't think it's working. I copied the test.exe to the \Temp folder and it still ran OK, will look at this, but probably not until late January. It looks like it's picking the Python in some other way (but not the PATH, that points to Python 2.7).

[Thomas Kluyver]

I wonder if once you hit C:\, any further .. elements are ignored. Then your ..\..\..\..\..\ just gets you back to C:\ from anywhere that's less than 6 folder levels deep.

[Vinay Sajip]

Yes, it looks like that's what's happening. Oh well.

[Vinay Sajip]

The obvious Win32 API to check for a relative path - PathIsRelative() - I believe returns FALSE for a path such as \test.exe - do we need to support drive-relative paths like this? I would assume not, for the use cases we've talked about - relative paths in shebangs should always start with '..\' for the wrapper to support them, right?

[Thomas Kluyver]

I don't have a use case for paths from the root of the current drive (\foo), but it might be useful to have a relative path that doesn't start with ..\, e.g. if Python is in a subfolder relative to your exe: subfolder\Python.exe.

It would also be fine to have a special marker for relative-to-exe paths like %EXEDIR%\subfolder\Python.exe, if that makes implementation easier.

[Adrien Ferrand]

What about staying to explicit absolute paths, or implicit relative paths against working directory ?

Here is my idea. Let's imagine that the wrapper as the ability to interpolate the given shebang against variables, in a really similar way than string interpolation with bash. Let also assume that the wrapper would have the ability to resolve its own absolute path in a variable launcher_dir and optionally the absolute path of working directory as working_dir (with value equals to launcher_dir if execution is outside a shell).

Then, we would have the following possible shebangs:

#
#!"C:\Python\python.exe" => explicit resolution against absolute path
#!"python.exe" => implicit resolution against available command in `PATH`
#!"${working_dir}\..\python.exe" => implicit resolution against relative path to the working directory
#!"${launcher_dir}\..\python.exe" => implicit resolution against relative path to the launcher directory

With the obvious advantage that this approach cannot break anything for existing shebangs: their behaviors will stay untouched. You will need the explicit ${} variable injection to trigger an interpolation. And the variables themselves are absolute directory to have a consistent behavior.

[Adrien Ferrand]

Well, similar idea to Thomas Kluyver in fact ^^

[Vinay Sajip]

Trying to keep it simple ... if the shebang is not "python.exe" and is a relative path according to the PathIsRelative() API just before dispatching, we can take the launcher executable's directory and combine that with the relative path to get the final executable. I have made some changes in the latest version to do this (search for SUPPORT_RELATIVE_PATH to see the changes), and the corresponding executables are available in the downloads section dated today. Try them out and let me know how you get on. By all means comment on the source changes to see if they look sensible.

I've done smoke testing with subfolders below the launcher and also parent folders, seems to work as expected.

[Thomas Kluyver]

I've had a look at the changes and they seem reasonable. I don't see a practical use for shebangs relative to the cwd, so interpreting relative paths as being from the launcher's own directory makes sense. I'll leave testing to people who have more convenient access to Windows than me.

One suggestion: instead of the hardcoded check for python.exe, why not check if the path has a \ in (or /, if you allow that). This avoids special-casing one particular name, and it lines up with how Unix shells find commands - that's why we have to type ./foo to run a script from the CWD, whereas foo searches PATH.

[Vinay Sajip]

The reason for the special casing of "python.exe" is that it is a relative path, but not one we want to process as meaning "next to the launcher executable". If we remove the special case check, I think other scripts that are out there will stop working, as people sometimes put just "python.exe" in shebangs and expect it to be picked up via the PATH. Note that a side-by-side python.exe could still be supported with this scheme, by using the shebang ".\python.exe". In fact, perhaps we should check for plain "pythonX[.Y].exe" as well.

[Adrien Ferrand]

Really sorry to bother you with this, but I am still concerned by the retro-compatibility breakage. Relatives paths were resolved against the current working directory, and with this evolution, compiled wrappers done the same way will see there fonctionnalities changed silently.

Even if the feature was not documented, with a project of such large audience, there could have some users that found it and relies on relative directories that are resolved against working directory.

It is imaginable that a complex application sets its working directory at runtime, and propagates it to any subprocess that will call wrappers. In particular python virtual environnements are portable, changing there paths make the all thing still working. So there is definitively a relative resolution. I think it is not done at the wrapper layer, but on its caller, but it is close to a breakage situation in my opinion.

I would do an explicit declaration of the relative resolution against the wrapper directory in shebangs. As it is a new functionality, users will adapt their code if they want it, instead of facing an unexpected functionnality shift.

Nevertheless I will test the current implementation on my Windows machine and report quickly.

[Thomas Kluyver]

@vinay.sajip I'm suggesting that you keep that check, but expand it to any shebang without a slash in - so not just python.exe and python3.exe, but any foo.exe would be looked up in PATH, and you'd have to use .\foo.exe to find it in the same directory as the launcher.

Regarding backward compatibility: I think it's up to Vinay. You're not bound to support every undocumented behaviour forever on a free project just in case someone is using it, and it would be a really weird use case.

[Adrien Ferrand]

Well, I will continue to be the devil's advocate. I also think it would be a real mistake to build something on a hidden and unsupported behavior, but well, we all made bad decision designs. More than that, in a such low level, intrications are potentially on several layers and can hit people that rely on libraries that relies on libraries with bad decision designs.

I am saying that because not latter than yesterday, I hit a compatibility breakage on a third library for certbot that was introduced (and documented) 10 years ago.

At least this breakage should be documented, and I will definitively test the new launchers against distutils, as it is a good entrypoint to see potential large intricated side-effects.

[Vinay Sajip]

I agree we have to move carefully because of the need to maintain backward compatibility. Breaking changes affect not only developers who take advantage of undocumented behaviour but all their users too, as Adrien has said. I don't have too much more time to spend on this now, I only managed to do what I did today because a client released me a little early.

[Robert Iannucci Jr.]

I would expect documentation to be enough; path relative to CWD would be quite difficult to use? Probably setting PATH is equivalently difficult in such scenarios (i.e. instead of doing cd python.exe\dir && path\to\script.exe do PATH=python.exe\dir;%PATH% && path\to\script.exe). But... yeah knowing for sure is hard :)

FWIW, I downloaded the new t32.exe and it works brilliantly ; I installed python to a folder, generated e.g. Scripts\pip.exe, then replaced pip.exe with (essentially) t32.exe++"#!..\python.exe\r\n"++tail_of_pip.exe. Invoking this works, and continues to work after renaming the folder.

It would probably be worth adding a blurb about this in ReadMe.txt too though, for the next adventurous wayfarer.

[Vinay Sajip]

How about this? Current released behaviour remains exactly as is, unless the shebang executable starts with the prefix <LD> (standing for Launcher Directory, interpreted case-insensitively), in which case the text following the prefix is interpreted as a launcher-directory-relative path. Ordinary relative paths and absolute paths needn't change. Note that < and > are illegal in Windows pathnames, and so should not be found in scripts in the wild. We could use pathname-illegal * or ? characters instead, if they have advantages. Adding @pmoore as he may be interested in this discussion.

[Thomas Kluyver]

So a shebang would look like #!<LD>subdir\python.exe? Or would it need a slash like #!<LD>\subdir\python.exe?

Any variant of this proposal would work nicely for Pynsist. :-)

[Adrien Ferrand]

And what about quoted paths? Because if the anchor is considered as a modifier for path interpretation, it should be #!<LD>"my relative path to\python.exe", but if it is equivalent to a string interpolation, then it should be #!"<LD>\my relative path to\python.exe".

Note also that after checks around Wikipedia, well informed sources in stackoverflow, and my own tests, bare relative shebangs in POSIX systems are indeed resolved against the working directory, not the script directory. So using explicit modifiers to interpret relative paths to the script directory is compatible with what you would expect in the POSIX world.

[Paul Moore]

Sounds like a reasonable option to me. I was originally more in favour of making launcher-relative be the default, but the comment made by @adferrand about POSIX behaviour makes sense.

It would be good to clarify how the <LD> prefix interacts with quoting.

One other point - from my (very brief) reading of the source, #!python will search PATH for the Python executable, which is not a relative filename search. So that's another wrinkle. It might be worthwhile documenting the exact rules, particularly now that they are starting to get quite complex. Reading the source code isn't a particularly easy way of working out what's intended :-(

[Vinay Sajip]

+1 to documenting things before going further - so that all the wrinkles can be laid bare and then hopefully ironed out,

[Robert Iannucci Jr.]

FWIW, <LD> prefix would be fine for me as well :). I wouldn't mind a more explicit string though, like <launcher_dir> or some such; I don't think saving the few bytes is all that necessary, and <launcher_dir> is probably more obvious to the uninitiated developer who stumbles on this later.

[Vinay Sajip]

Documentation for how it could work is here. Comments welcome.

[Paul Moore]

I got an error "You do not have access to this wiki". Does it need to be made public?

[Vinay Sajip]

How odd. It was private, but that is supposed to be such that "Anyone with repository access can view the wiki. " I've made it public now.

[Paul Moore]

Two examples there:

• python.exe -> python.exe
• foo.exe -> foo.exe

are unclear. Is the resulting value searched for in PATH, or is it expected to exist as a file in the current directory?

Otherwise, it looks OK to me.

[Thomas Kluyver]

The proposal looks good to me; thanks Vinay. I agree with Paul that it would be good to clarify what a plain foo.exe shebang actually does, even if it's not your code that deals with it.

Is there any downside to quoting the resulting path? Is it simpler to always quote the resulting interpreter path, rather than only doing it when there are spaces? That would seem more consistent.

[Vinay Sajip]

I've updated the Wiki page to clarify the first four cases in the bullet-point list.

[Thomas Kluyver]

I don't want to pester you, but I'd still be interested in this feature as and when it gets into a released version of distlib.

[Vinay Sajip]

Sure - I just have my hands fairly full right now, and this might be the case until mid-March. I'll do what I can. Thanks for being patient!

[Thomas Kluyver]

Thanks Vinay! There's no rush from my side - I have something that works at present, so this is a nice-to-have feature.

[Vinay Sajip]

Changes in c6b0f63 implement as discussed in the Wiki page. The launchers in the Downloads section have been updated. Please try out and report back!

[Thomas Kluyver]

Thanks Vinay! Do you have any plans to put the new launchers in distlib?

I'll try to find time to test them, but as I'm not using Windows myself, it's not super straightforward. So I can't promise when it will be.

[Vinay Sajip]

Perhaps you have a friendly Pynsist user who could help out. Of course, I do plan to update distlib with these launchers, but I don't have a specific time frame for that (in terms of doing a new release - there are a few other things that are higher in my priority list).