Possible to have launcher exe find python.exe relative to itself?
(Repost from https://bitbucket.org/pypa/distlib/issues/117)
The scenario I'm facing is that I'd like to be able to distribute python in a 'relocatable' form (think: install to a thumbdrive).
The python interpreter itself seems pretty happy with this arrangement, but the .exe's generated in Scripts (IIUC these are a shim script cated onto the end of one of the launcher .exes in distlib) seem to only support an absolute path to the python interpreter, or a bare exe name (like "python.exe"). The absolute path won't work well for a relocatable distribution for obvious reasons :). Setting it to "python.exe" works slightly better... but it requires that python.exe is on %PATH%, which is not something I can guarantee easily.
Furthermore, I have an ollld python distribution hacked up by one of my teammates (some version of 2.7.6) still using the non-appended archives (i.e. foo.exe launches foo-script.py), and it appears that those scripts begin with #!python.exe and the .exe can find the correct interpreter (likely relative to the .exe launcher)!
It seems that some version of the launcher used to be able to find python.exe relative to itself, but I can't find evidence of this feature in any of the available sources (i.e. anything added to distlib after launcher.c was available).
So, is it possible that this feature: 1) did actually exist in a previous version of the launcher 2) could come back somehow? (I'd be willing to submit a patch if need be :)). Maybe something like "%cd%..\python.exe" so that this behavior doesn't happen magically for "python.exe"?
Comments (35)
-
repo owner -
I haven't tried this, but are we confident it's working relative to the folder containing the exe rather than the cwd? I would assume that unless you write specific code for this, relative paths are interpreted from the cwd.
-
repo owner I had tried from another folder,
which seemed to be OK, but actually I don't think it's working. I copied the test.exe to the \Temp folder and it still ran OK, will look at this, but probably not until late January. It looks like it's picking the Python in some other way (but not the
PATH
, that points to Python 2.7). -
I wonder if once you hit
C:\
, any further..
elements are ignored. Then your..\..\..\..\..\
just gets you back toC:\
from anywhere that's less than 6 folder levels deep. -
repo owner Yes, it looks like that's what's happening. Oh well.
-
repo owner The obvious Win32 API to check for a relative path -
PathIsRelative()
- I believe returnsFALSE
for a path such as\test.exe
- do we need to support drive-relative paths like this? I would assume not, for the use cases we've talked about - relative paths in shebangs should always start with '..\' for the wrapper to support them, right? -
I don't have a use case for paths from the root of the current drive (
\foo
), but it might be useful to have a relative path that doesn't start with..\
, e.g. if Python is in a subfolder relative to your exe:subfolder\Python.exe
.It would also be fine to have a special marker for relative-to-exe paths like
%EXEDIR%\subfolder\Python.exe
, if that makes implementation easier. -
What about staying to explicit absolute paths, or implicit relative paths against working directory ?
Here is my idea. Let's imagine that the wrapper as the ability to interpolate the given shebang against variables, in a really similar way than string interpolation with bash. Let also assume that the wrapper would have the ability to resolve its own absolute path in a variable
launcher_dir
and optionally the absolute path of working directory asworking_dir
(with value equals tolauncher_dir
if execution is outside a shell).Then, we would have the following possible shebangs:
# #!"C:\Python\python.exe" => explicit resolution against absolute path #!"python.exe" => implicit resolution against available command in `PATH` #!"${working_dir}\..\python.exe" => implicit resolution against relative path to the working directory #!"${launcher_dir}\..\python.exe" => implicit resolution against relative path to the launcher directory
With the obvious advantage that this approach cannot break anything for existing shebangs: their behaviors will stay untouched. You will need the explicit
${}
variable injection to trigger an interpolation. And the variables themselves are absolute directory to have a consistent behavior. -
Well, similar idea to Thomas Kluyver in fact ^^
-
repo owner Trying to keep it simple ... if the shebang is not "python.exe" and is a relative path according to the
PathIsRelative()
API just before dispatching, we can take the launcher executable's directory and combine that with the relative path to get the final executable. I have made some changes in the latest version to do this (search forSUPPORT_RELATIVE_PATH
to see the changes), and the corresponding executables are available in the downloads section dated today. Try them out and let me know how you get on. By all means comment on the source changes to see if they look sensible.I've done smoke testing with subfolders below the launcher and also parent folders, seems to work as expected.
-
I've had a look at the changes and they seem reasonable. I don't see a practical use for shebangs relative to the cwd, so interpreting relative paths as being from the launcher's own directory makes sense. I'll leave testing to people who have more convenient access to Windows than me.
One suggestion: instead of the hardcoded check for
python.exe
, why not check if the path has a\
in (or/
, if you allow that). This avoids special-casing one particular name, and it lines up with how Unix shells find commands - that's why we have to type./foo
to run a script from the CWD, whereasfoo
searches PATH. -
repo owner The reason for the special casing of "python.exe" is that it is a relative path, but not one we want to process as meaning "next to the launcher executable". If we remove the special case check, I think other scripts that are out there will stop working, as people sometimes put just "python.exe" in shebangs and expect it to be picked up via the
PATH
. Note that a side-by-sidepython.exe
could still be supported with this scheme, by using the shebang ".\python.exe". In fact, perhaps we should check for plain "pythonX[.Y].exe" as well. -
Really sorry to bother you with this, but I am still concerned by the retro-compatibility breakage. Relatives paths were resolved against the current working directory, and with this evolution, compiled wrappers done the same way will see there fonctionnalities changed silently.
Even if the feature was not documented, with a project of such large audience, there could have some users that found it and relies on relative directories that are resolved against working directory.
It is imaginable that a complex application sets its working directory at runtime, and propagates it to any subprocess that will call wrappers. In particular python virtual environnements are portable, changing there paths make the all thing still working. So there is definitively a relative resolution. I think it is not done at the wrapper layer, but on its caller, but it is close to a breakage situation in my opinion.
I would do an explicit declaration of the relative resolution against the wrapper directory in shebangs. As it is a new functionality, users will adapt their code if they want it, instead of facing an unexpected functionnality shift.
Nevertheless I will test the current implementation on my Windows machine and report quickly.
-
@vinay.sajip I'm suggesting that you keep that check, but expand it to any shebang without a slash in - so not just
python.exe
andpython3.exe
, but anyfoo.exe
would be looked up in PATH, and you'd have to use.\foo.exe
to find it in the same directory as the launcher.Regarding backward compatibility: I think it's up to Vinay. You're not bound to support every undocumented behaviour forever on a free project just in case someone is using it, and it would be a really weird use case.
-
Well, I will continue to be the devil's advocate. I also think it would be a real mistake to build something on a hidden and unsupported behavior, but well, we all made bad decision designs. More than that, in a such low level, intrications are potentially on several layers and can hit people that rely on libraries that relies on libraries with bad decision designs.
I am saying that because not latter than yesterday, I hit a compatibility breakage on a third library for certbot that was introduced (and documented) 10 years ago.
At least this breakage should be documented, and I will definitively test the new launchers against distutils, as it is a good entrypoint to see potential large intricated side-effects.
-
repo owner I agree we have to move carefully because of the need to maintain backward compatibility. Breaking changes affect not only developers who take advantage of undocumented behaviour but all their users too, as Adrien has said. I don't have too much more time to spend on this now, I only managed to do what I did today because a client released me a little early.
-
reporter I would expect documentation to be enough; path relative to CWD would be quite difficult to use? Probably setting PATH is equivalently difficult in such scenarios (i.e. instead of doing
cd python.exe\dir && path\to\script.exe
doPATH=python.exe\dir;%PATH% && path\to\script.exe
). But... yeah knowing for sure is hard :)FWIW, I downloaded the new t32.exe and it works brilliantly ; I installed python to a folder, generated e.g. Scripts\pip.exe, then replaced pip.exe with (essentially) t32.exe++"#!..\python.exe\r\n"++tail_of_pip.exe. Invoking this works, and continues to work after renaming the folder.
It would probably be worth adding a blurb about this in ReadMe.txt too though, for the next adventurous wayfarer.
-
repo owner How about this? Current released behaviour remains exactly as is, unless the shebang executable starts with the prefix
<LD>
(standing for Launcher Directory, interpreted case-insensitively), in which case the text following the prefix is interpreted as a launcher-directory-relative path. Ordinary relative paths and absolute paths needn't change. Note that<
and>
are illegal in Windows pathnames, and so should not be found in scripts in the wild. We could use pathname-illegal*
or?
characters instead, if they have advantages. Adding @pmoore as he may be interested in this discussion. -
So a shebang would look like
#!<LD>subdir\python.exe
? Or would it need a slash like#!<LD>\subdir\python.exe
?Any variant of this proposal would work nicely for Pynsist. :-)
-
And what about quoted paths? Because if the anchor is considered as a modifier for path interpretation, it should be
#!<LD>"my relative path to\python.exe"
, but if it is equivalent to a string interpolation, then it should be#!"<LD>\my relative path to\python.exe"
.Note also that after checks around Wikipedia, well informed sources in stackoverflow, and my own tests, bare relative shebangs in POSIX systems are indeed resolved against the working directory, not the script directory. So using explicit modifiers to interpret relative paths to the script directory is compatible with what you would expect in the POSIX world.
-
Sounds like a reasonable option to me. I was originally more in favour of making launcher-relative be the default, but the comment made by @adferrand about POSIX behaviour makes sense.
It would be good to clarify how the
<LD>
prefix interacts with quoting.One other point - from my (very brief) reading of the source,
#!python
will searchPATH
for the Python executable, which is not a relative filename search. So that's another wrinkle. It might be worthwhile documenting the exact rules, particularly now that they are starting to get quite complex. Reading the source code isn't a particularly easy way of working out what's intended :-( -
repo owner +1 to documenting things before going further - so that all the wrinkles can be laid bare and then hopefully ironed out,
-
reporter FWIW, <LD> prefix would be fine for me as well :). I wouldn't mind a more explicit string though, like <launcher_dir> or some such; I don't think saving the few bytes is all that necessary, and <launcher_dir> is probably more obvious to the uninitiated developer who stumbles on this later.
-
repo owner Documentation for how it could work is here. Comments welcome.
-
I got an error "You do not have access to this wiki". Does it need to be made public?
-
repo owner How odd. It was private, but that is supposed to be such that "Anyone with repository access can view the wiki. " I've made it public now.
-
Two examples there:
python.exe
->python.exe
foo.exe
->foo.exe
are unclear. Is the resulting value searched for in PATH, or is it expected to exist as a file in the current directory?
Otherwise, it looks OK to me.
-
The proposal looks good to me; thanks Vinay. I agree with Paul that it would be good to clarify what a plain
foo.exe
shebang actually does, even if it's not your code that deals with it.Is there any downside to quoting the resulting path? Is it simpler to always quote the resulting interpreter path, rather than only doing it when there are spaces? That would seem more consistent.
-
repo owner I've updated the Wiki page to clarify the first four cases in the bullet-point list.
-
I don't want to pester you, but I'd still be interested in this feature as and when it gets into a released version of distlib.
-
repo owner Sure - I just have my hands fairly full right now, and this might be the case until mid-March. I'll do what I can. Thanks for being patient!
-
Thanks Vinay! There's no rush from my side - I have something that works at present, so this is a nice-to-have feature.
-
repo owner - changed status to resolved
Changes in c6b0f63 implement as discussed in the Wiki page. The launchers in the Downloads section have been updated. Please try out and report back!
-
Thanks Vinay! Do you have any plans to put the new launchers in distlib?
I'll try to find time to test them, but as I'm not using Windows myself, it's not super straightforward. So I can't promise when it will be.
-
repo owner Perhaps you have a friendly Pynsist user who could help out. Of course, I do plan to update
distlib
with these launchers, but I don't have a specific time frame for that (in terms of doing a new release - there are a few other things that are higher in my priority list). - Log in to comment
This might actually be working already, without me doing anything special. Just to test the current behaviour, I made a change to the test script
make_test.py
which combines the command-line executable with a short inline script. I then invoked this with a relative shebang, and it appears to work as expected. See the screenshot:So, you might want to try this and report back any specific problem you find.