Python launcher invoked with command "py" fails to start process

[kai2nenobu]

Bug report

When I invoke a python launcher of version 3.11.0 with "py" command, a python launcher fails to start process.

Step to reproduce

1. Install python 3.11.0 with py launcher for all users.

2. Invoke "py" in command prompt.

3. I expect to launch a python interpreter, but got a error as below

   >"py"
   Unable to create process using 'C:\Python311\py"':  The system cannot find the file specified.

   Debug output when enable PYLAUNCHER_DEBUG

   >"py"
   argv0: py
   version: 3.11.0
   SearchInfo.originalCmdLine: "py"
   SearchInfo.restOfCmdLine:
   SearchInfo.executablePath: (null)
   SearchInfo.scriptFile: (null)
   SearchInfo.executable: py"
   SearchInfo.executableArgs: (null)
   SearchInfo.company: (null)
   SearchInfo.tag: (null)
   SearchInfo.oldStyleTag: False
   SearchInfo.lowPriorityTag: False
   SearchInfo.allowDefaults: False
   SearchInfo.allowExecutableOverride: False
   SearchInfo.windowed: False
   SearchInfo.list: False
   SearchInfo.listPaths: False
   SearchInfo.help: False
    -V:3.11          C:\Python311\python.exe
    -V:3.10          C:\Python310\python.exe
    -V:3.9           C:\Python39\python.exe
    -V:3.7           C:\Python37\python.exe
   env.company: PythonCore
   env.tag: 3.11
   # about to run: C:\Python311\py"
   Unable to create process using 'C:\Python311\py"':  The system cannot find the file specified.

I think this error seems to be similar to #95285. If I invoke py or "py.exe", a python interpreter starts well as I expected.

>py
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

>"py.exe"
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

In version 3.10.8, all of "py", py and "py.exe" start a python interpreter correctly.

Your environment

• CPython versions tested on: 3.11.0
• Operating system and architecture: Windows 10 (10.0.19045.2310)/amd64

Linked PRs

• gh-99442: Fix handling in py.exe launcher when argv[0] does not include a file extension #99542
• [3.11] gh-99442: Fix handling in py.exe launcher when argv[0] does not include a file extension (GH-99542) #99579

[kai2nenobu]

This error causes a failure to build MSI installer between version 3.7 and 3.10. (Example of GitHub Actions build failure in my personal project).

This is because WiX invokes "py" -m pip install -U blurb during build for exe.wixproj and recent GitHub-Hosted runner uses Python Launcher of version 3.11.0.

[pfmoore]

Confirmed, I am seeing the same issue (see the linked rust issue report above).

Command: "py" "-V"
argv0: py
version: 3.11.0
SearchInfo.originalCmdLine: "py" -V
SearchInfo.restOfCmdLine:  -V
SearchInfo.executablePath: (null)
SearchInfo.scriptFile: (null)
SearchInfo.executable: py"
SearchInfo.executableArgs: (null)
SearchInfo.company: (null)
SearchInfo.tag: (null)
SearchInfo.oldStyleTag: False
SearchInfo.lowPriorityTag: False
SearchInfo.allowDefaults: False
SearchInfo.allowExecutableOverride: False
SearchInfo.windowed: False
SearchInfo.list: False
SearchInfo.listPaths: False
SearchInfo.help: False
 -V:3.11          C:\Users\Gustav\AppData\Local\Programs\Python\Python311\python.exe
 -V:3.10          C:\Users\Gustav\AppData\Local\Programs\Python\Python310\python.exe
 -V:3.9           C:\Users\Gustav\AppData\Local\Programs\Python\Python39\python.exe
 -V:3.8           C:\Users\Gustav\AppData\Local\Programs\Python\Python38\python.exe
env.company: PythonCore
env.tag: 3.11
# about to run: C:\Users\Gustav\AppData\Local\Programs\Python\Python311\py" -V
Unable to create process using 'C:\Users\Gustav\AppData\Local\Programs\Python\Python311\py" -V': The system cannot find the file specified.

[eryksun]

In parseCommandLine(), the value of end points to just past the closing double quote if there's no "." found in the tail component. findArgumentEnd() could be modified to exclude the double quote from the end, since the double quote at the end isn't actually part of the command-line argument. It's a syntax character for argument parsing.

cpython/PC/launcher2.c

Lines 565 to 580 in 88385b8

	const wchar_t *tail = findArgumentEnd(search->originalCmdLine, -1);
	const wchar_t *end = tail;
	search->restOfCmdLine = tail;
	while (--tail != search->originalCmdLine) {
	if (*tail == L'.' && end == search->restOfCmdLine) {
	end = tail;
	} else if (*tail == L'\\' || *tail == L'/') {
	++tail;
	break;
	}
	}
	if (tail == search->originalCmdLine && tail[0] == L'"') {
	++tail;
	}
	// Without special cases, we can now fill in the search struct
	int tailLen = (int)(end ? (end - tail) : wcsnlen_s(tail, MAXLEN));

---

A couple of issues in findArgumentLength(), which could also be resolved here without creating a new issue for them:

• If the buffer doesn't start with a double quote, it needs to search for white space -- at least tab ("\t") in addition to space (" ").
• Since findArgumentLength() is used by findArgumentEnd(), which is only ever called to parse the first argument of the command line, it should not implement backslash escaping of double quotes. When parsing the first argument of the command line, both CreateProcessW() and CommandLineToArgvW() enter quote mode if the first character is a double quote, in which case the argument ends at the next double quote, regardless of whether it's followed by whitespace. That's a bit simpler than the C runtime, but even the C runtime doesn't implement escaping of double quotes in the first argument.

[pfmoore]

The fix appears to have addressed part of my issue, but I'm still getting the same error in a slightly different case. I haven't yet been able to reduce the test case to something smaller, but if I use the just command line utility, with a justfile as follows:

set windows-shell := ["pwsh.exe", "-NoLogo", "-Command"]

test_py:
    #!py
    print(12)

then running just test_py gives the error:

Unable to create process using 'C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\py C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\test_py': The system cannot find the file specified.

If I add some debugging prints, I can see that the command being run was "py" "C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\test_py".

[zooba]

Thanks for the quick extra test! This one looks like it's probably shebang parsing - can you include the whole output you get with PYLAUNCHER_DEBUG set?

[zooba]

Oh, this is probably the deliberate change of behaviour to make shebang lines only search PATH if you use /usr/bin/env in them.

If it falls through to the default case, we don't let you search for arbitrary executables. It will only look adjacent to the script file. The old behaviour was never documented, and isn't safe (the new behaviour is only mildly safer, but at least you can scan for usr/bin/env and recognise when it's going to happen, rather than trying to prove a negative). The new documentation reads:

Shebang lines that do not match any of these patterns are treated as Windows paths that are absolute or relative to the directory containing the script file. This is a convenience for Windows-only scripts, such as those generated by an installer, since the behavior is not compatible with Unix-style shells. These paths may be quoted, and may include multiple arguments, after which the path to the script and any additional arguments will be appended.

[eryksun]

#!py is a strange shebang, asking the launcher to recursively run itself. There should be a check in _useShebangAsExecutable() to prevent recursively running the launcher, as you did for the "#!/usr/bin/env" case in searchPath(). Or modify checkShebang() to always check for this. Otherwise #!py will blow up if the launcher is named "py.exe", and the script is beside it.

[pfmoore]

#!py is a strange shebang, asking the launcher to recursively run itself.

It is, and I don't love it, but most other approaches for my use case have downsides. I will say that the old launcher went into a hard recursive loop on this, locking my machine with masses of processes. That's extremely bad, and avoiding that is definitely worthwhile. Obviously, from what I'm doing here, you can tell that I'd prefer #!py to be treated the same as no shebang (i.e. run the script with Python)... And indeed, that's what apparently happens in a normal script. If I have test.py containing

#!py
print("Hello")

and I run it as py test.py, then it does indeed print "Hello"... So all I'm really asking is that the same behaviour works in the case we're triggering here (where the command line has "py" quoted).

The reason I'm doing this at all is that I'm using just, which lets you define actions starting with #! which it processes as a shebang. Just's processing is built in, and (ignoring details) it looks for the executable on PATH. So this shebang will cause just to run the action as a script using py.exe. But then, py re-interprets the shebang, leading to this issue.

The problem is that I don't want to hard-code a path for the interpreter in the shebang line, and the only way to run Python that's guaranteed to be on PATH is via py.exe.

I'm not even sure who I should be asking for help with this. In isolation, the behaviour of both just and py are reasonable. And if there was a python command on PATH, I'd use that. But the combination leaves no really good options.

[eryksun]

The new launcher resolves #!py relative to the script directory. Currently, this leads to an endless loop if the script is in the same directory as the "py.exe" launcher, but generally that's not the case. On Unix, #!py is resolved relative to the current working directory instead of the script directory.

Does just support passing command-line options in the shebang? If the shebang is #!py -3, then the launcher started with -3 won't read the shebang.

Alternatively, does just support Unix templates such as #!/usr/bin/env py? The launcher's implementation of the "env" command will search PATH, but it will disregard the shebang if the search resolves to itself.

---

On a related note, the launcher supports user-defined commands such as "py" in the [commands] section of "py.ini". The new launcher requires user-defined commands in a shebang to be qualified by one of the Unix template paths: "/usr/bin/env ", "/usr/bin/", or "/usr/local/bin/". A bare command is only supported if it starts with "python", which, among other possibilities, can override a registered installation for a "python[x[.y]]" command. With the old launcher, shebang commands that started with "python" were always reserved for registered installations, but user-defined commands that didn't start with "python" could be used without qualification, such as #!py instead of #!/usr/bin/py. I think the new semantics are an improvement, overall.

[pfmoore]

All of those are great ideas, thanks! I'll give them a try when I have some free time.

Overall, it does seem a bit non-obvious to find something that works, which is a shame, but I don't think there's an easy resolution for that. Long-term, we should ideally have a unified way of running Python on all platforms, but I doubt that will happen soon...

[pfmoore]

Actually, I had a few moments. Just's implementation of #!/usr/bin/env is problematic, because it relies on Cygwin (specifically the cygpath command) being available. That's not something I can rely on.

But using #!py -3 works fine, so that seems like an ideal fix. I'll submit a documentation request to the just project to suggest that form when creating recipes defined in Python on Windows (I can't think of a cross-platform way that will work, unfortunately).

[pfmoore]

And indeed, that's what apparently happens in a normal script.

Actually, it appears that the fix for this issue has broken that case 🙁

At this point, I'm frustrated by this whole thing. The new behaviour (treating the shebang as relative to the script location) is a change from the behaviour described in PEP 397:

Otherwise the command is assumed to be directly ready to execute - ie. a fully-qualified path (or a reference to an executable on the PATH) optionally followed by arguments.

which was (as far as I am aware) introduced without discussion, and which is not included in the "What's new in 3.11" documentation. If I'm wrong, please point me to the discussion of this new behaviour. It's quite possible that I would have approved the change if I'd been asked about it in advance, and I'm definitely not suggesting that the new behaviour is wrong as such - but I do think that we should have provided better support and information for people who might have been relying on the old behaviour.

Now that this has already happened, I'm not sure what, if anything, should be done. My feeling is that an "after the fact" update to the what's new document, offering migration advice in the "Porting to 3.11" section for people who were using the old behaviour, is probably the best thing to do. But I'm open to other suggestions (although I'm not keen on "do nothing", as you might have guessed 🙂).

[zooba]

Yeah, there were certainly no updates made to the PEP, and we based the changes on what was in the documentation (worth noting that the PEP has a number of inconsistencies in this area, and is also dated with respect to what's considered secure practices).

The discussion unfortunately seems to be in #98732, hidden under resolved comments (one of the reasons I always disliked the plan of moving everything to GitHub, but I didn't do this on purpose ;) ). FWIW, I found this by looking at NEWS for "launcher" and going to the one that was pretty clearly about changing the shebang handling.

I still think it's highly defensible to drop the CWD search, but happy to argue about dropping the PATH search for unqualified names. There are already fixes coming in 3.11.1 to get closer to old behaviour, so no harm in bringing back more. Let's do it on a new issue though.

At this point, I'm frustrated by this whole thing.

My sincere apologies for this. I know it's hard to keep up with all the changes going on in all the different places, but it's equally hard to know whose opinion ought to be sought on every issue. It's also frustrating for me to find that every change I make frustrates someone else for not being consulted (you're not the first), even if the change was accidental ;) , but I'm not sure what the solution is other than to stop contributing. 🤷‍♂️ That would be a childish response, but so would be pinging everyone on every discussion and waiting for quorum before doing anything. I'm open to ideas, and happy to discuss offline (with Paul, that is, not any random person who happens by here).

[timabroad]

But using #!py -3 works fine, so that seems like an ideal fix. I'll submit a documentation request to the just project to suggest that form when creating recipes defined in Python on Windows (I can't think of a cross-platform way that will work, unfortunately).

Is this still expected to work? I've got some old scripts that did this (and worked in the past), but don't seem to work on 3.11.8:

$ pwd
/c/test

$ cat foo.py
#!py -3
print('Hello, world!')

$ ./foo.py
Hello, world!

$ py foo.py
Unable to create process using 'C:\test\py -3 foo.py': The system cannot find the file specified.

In this case the launcher seems to be resolving the shebang as if it's a relative path to some executable in the current working directory.

[eryksun]

@timabroad

If you explain your use case for using the shebang "!#py -3", I may be able to suggest an alternative.

As is, the launcher tries to resolve "py" in the shebang relative to the script directory. It doesn't search the current working directory or PATH. On the other hand, "./foo.py" works because the shell that you're using parses the shebang and apparently searches PATH to find "py".

To search PATH for "py", the launcher requires the use of "#!/usr/bin/env py -3". However, that won't exactly work either if the search finds the path of the executing launcher. It will just ignore the shebang and execute the script using the default version of Python in order to avoid problems with recursive execution of the launcher. The "-3" command-line option in the shebang avoids the recursive execution problem, but the implementation of the search isn't designed to look for that as an override.

[timabroad]

Ah, so probably #!/usr/bin/env py -3 is what I'm after then. I'm just tring to get the script to work with either invocation (./foo.py or py foo.py). The -3 is probably somewhat irrelevant - if the user is directly calling py foo.py then I assume they want to control which version of Python to run, but if it's run directly with ./foo.py then I just wanted to make sure it failed early if you only had python 2 available (the script is old enought that that was a concern at the time 😛).

Anyway, searching for the error only really brought me here plus a couple of other tickets, but this one had a comment explicity suggesting that what I had should work as of quite recently. (Hopefully this comment is then enough to help others running into the same issue 😃).

[zooba]

We probably need to add "py" as a special case that restarts the whole search process with a new command line but with shebangs disabled.

The earlier fix corrected handling for unnecessary quotes, but may as well reopen to further add a special case for using py instead of the more portable #!/usr/bin/python3 (which would be the equivalent of #!py -3).

[zooba]

Unfortunately, other people demanded that usr/bin/env python3 should prioritise the PATH search over the version number, so we can't have perfect compatibility that way. But perhaps all these py -3 cases would be quite happy with a python3 on PATH override?

[eryksun]

if it's run directly with ./foo.py then I just wanted to make sure it failed early if you only had python 2 available (the script is old enought that that was a concern at the time 😛).

Note that if CMD or PowerShell is used instead of bash, then directly running a .py script will execute the associated command-line template for .py files, which is usually the "py.exe" launcher. If so, the launcher's implementation of "#!/usr/bin/env py" will search for "py.exe" in PATH, find itself, and then opt to ignore the shebang and run the default Python. In most cases that should be 3.x, unless the user has configured 2.x as the default via PY_PYTHON or "py.ini".

[eryksun]

Ah, so probably #!/usr/bin/env py -3 is what I'm after then

I just checked this using the MSYS2 version of bash and env. There's a snag. bash passes "py -3" as a single argument to "/usr/bin/env", which in turn requires the option "-S" to split this string into multiple command-line arguments. The launcher's implementation could support "#!/usr/bin/env -S", basically by just consuming the "-S", but currently it doesn't.

[zooba]

There's a limit on how much POSIX emulation we want to support, and I think -S is going too far. Especially since py -3 isn't portable anyway - /usr/bin/env python3 is the portable one, with /usr/bin/python3 and /usr/local/bin/python3 as valid though apparently less portable alternatives (and any of those can have a full tag specified on the end, so /usr/bin/python3.13-arm64 if you really want).

Any other paths not matching those templates can be added to the [commands] section in py.ini alongside the py.exe or in the user's %LocalAppData%\py.ini (which is not strictly the right place to put it, but historical reasons win). We don't need to have them all built in.

[eryksun]

Unfortunately, other people demanded that usr/bin/env python3 should prioritise the PATH search over the version number, so we can't have perfect compatibility that way. But perhaps all these py -3 cases would be quite happy with a python3 on PATH override?

The launcher's "env" implementation falls back on a virtual "python[X[.Y[.exe]]]" command if the executable name can't be found on PATH. For example:

argv0: C:\Windows\py.exe
version: 3.12.2
# Read 57 bytes from C:\Temp\test.py to find shebang line
Shebang: /usr/bin/env python3
# Did not find python3.exe on PATH
# Reading from C:\Users\user\AppData\Local\py.ini for commands/python3
# Did not find file C:\Users\user\AppData\Local\py.ini
# Reading from C:\Windows\py.ini for commands/python3
# Did not find file C:\Windows\py.ini
# Treating shebang command '3' as 'py -3'

but may as well reopen to further add a special case for using py instead of the more portable #!/usr/bin/python3 (which would be the equivalent of #!py -3)

"py[.exe] [-X[.Y]]" could be reserved as a virtual command like "python[.X[.Y[.exe]]]". Then handling "#!/usr/bin/env py [-X[.Y]]" could default to a virtual command if either "py" isn't found or if the launcher finds itself.

[timabroad]

And just for reference, the reason I'm using py (or py -3) instead of python3 is that that's all I have on my PATH:

$ which python
which: no python in ...

$ which python3
which: no python3 in ...

Note that if CMD or PowerShell is used instead of bash, then directly running a .py script will execute the associated command-line template for .py files, which is usually the "py.exe" launcher. If so, the launcher's implementation of "#!/usr/bin/env py" will search for "py.exe" in PATH, find itself, and then opt to ignore the shebang and run the default Python. In most cases that should be 3.x, unless the user has configured 2.x as the default via PY_PYTHON or "py.ini".

That's a good point. So perhaps the logic (if finding itself in the shebang line) needs to be:

1. If no selection criteria were used by the current invocation of py then parse the shebang line for selection criteria (eg -3 or whatever).
2. If the user specified any selection criteria then ignore the shebang line and go with what the user selected.

Or maybe the second case should merge the criteria and flag an error if there's a conflict - that is, if the user specifies py -3 script.py and the script says #!py -3.8 then it should probably pick 3.8 rather than 3.11. Maybe with a --ignore-shebang option or something if you really insist on running it with py -2. (I'm assuming this will add complication to the implementation though, as it presumably doesn't have any way to combine criteria currently?)

[timabroad]

There's a limit on how much POSIX emulation we want to support, and I think -S is going too far. Especially since py -3 isn't portable anyway - /usr/bin/env python3 is the portable one, with /usr/bin/python3 and /usr/local/bin/python3 as valid though apparently less portable alternatives (and any of those can have a full tag specified on the end, so /usr/bin/python3.13-arm64 if you really want).

But even /usr/bin/env python3 isn't portable to bash on Windows, by default at least, since I don't think Python adds itself to PATH by default (at least my setup doesn't have it, and I don't think I'd have explicitly unset it). So you'd have to explicitly add it (either in Windows - for which I think there's an option in the installer - or in your .bashrc or something) to make that work there. I guess you could also add a script on your bash PATH called python3 that forwards its arguments to py -3, but again, it won't happen out of the box.

Has anyone ever considered/suggested porting py to non-Windows platforms? That would potentially allow py to (eventually) become the cross-platform way to start a Python script... although to mix arguments like -3 with using /usr/bin/env you'd need to use #!/usr/bin/env -S py -3, but I don't think -S is POSIX-specified 🤷‍♂️.

[zooba]

So perhaps the logic (if finding itself in the shebang line) needs to be:

The logic you describe is basically what's already in place, though it's way more complicated because there are 4-5 different ways to set defaults. Shebangs show up somewhere in the middle.

I'm keen to not make this any more complicated, which is why about the only logic I'd consider is detecting py in the shebang and restarting processing with a new command line (and probably explicitly suppressing shebang handling, since we can't be sure that the original shebang line actually specified an option that would suppress it).

Has anyone ever considered/suggested porting py to non-Windows platforms?

Yes, there is also a py command that is similar. However, its implementation is vastly different, and its roadmap is very different. It's not part of the core project, so there's not necessarily any alignment, and we can't direct its behaviour.

Besides, one major difference is that it doesn't do shebangs at all, because it relies on the system to handle them. If you py script-with-shebang.py with that one, you get the default py behaviour. The shebang handling really is a hack to help out Windows users running scripts that were originally intended for POSIX.

But even /usr/bin/env python3 isn't portable to bash on Windows, by default at least, since I don't think Python adds itself to PATH by default

It depends exactly which emulation you're in. Cygwin brings its own Python build, which should include a python3, and if you installed Python from the Windows Store then you'll get a python3 on PATH (though last I heard, Cygwin and anything based on it - which seems to be every Windows Bash implementation - isn't capable of launching Store apps, so that's not much help). The environment is different enough and far enough out of our control that we don't support it, though we try not to do anything to deliberately break it.

[eryksun]

Or maybe the second case should merge the criteria and flag an error if there's a conflict - that is, if the user specifies py -3 script.py and the script says #!py -3.8 then it should probably pick 3.8 rather than 3.11.

I prefer the current behavior, which ignores the shebang if the launcher is executed with a "-X[.Y][-32|-64]" or "-V:tag" command-line option.

But even /usr/bin/env python3 isn't portable to bash on Windows, by default at least, since I don't think Python adds itself to PATH by default

Yes, there are issues for POSIX environments on Windows that have to be resolved manually. For Windows, with the shebang "#!/usr/bin/env python[X[.Y][-32|-64]][.exe]", if the launcher can't find the executable in PATH, it instead handles the virtual "python[X[.Y][-32|-64]]" command. Except I just noticed the following bug in the code that ignores ".exe", given the shebang "#!/usr/bin/env python.exe":

argv0: C:\Windows\py.exe
version: 3.12.2
# Read 60 bytes from C:\Temp\test.py to find shebang line
Shebang: /usr/bin/env python.exe
# Did not find python.exe on PATH
# Reading from C:\Users\user\AppData\Local\py.ini for commands/python.exe
# Did not find file C:\Users\user\AppData\Local\py.ini
# Reading from C:\Windows\py.ini for commands/python.exe
# Did not find file C:\Windows\py.ini
# Treating shebang command '.exe' as 'py -.exe'
# Cannot select defaults for tag '.exe'

[zooba]

I just noticed the following bug in the code that ignores ".exe", given the shebang "#!/usr/bin/env python.exe":

Whoops, yeah, that's a bug.

But at the same time, the intent of this limited shebang support isn't to provide an equivalent to what POSIX shells provide. It is really only meant to be limited to a few options that are compatible with other OSs so that developers who want to be compatible between both are able to achieve it.

I've chatted with Brett, who maintains the POSIX launcher, and he said that the shebang handling in that one is entirely limited to the same basic templates as on Windows in order to restrict Python version. Windows has to handle a few more features, but I'm pretty strongly leaning towards narrowing the scope back to what seems to have been the original intent: basic compatibility with POSIX shells.

What this would mean is that any shebang that doesn't work on a typical/reasonable Linux system is out of scope, even on Windows, and is unsupported. It doesn't mean that any shebang that does work there will be supported, but that the intent of shebang parsing is primarily to infer the -V:x.y argument and nothing more. It also doesn't mean that existing support gets immediately removed, but it does draw the line against additional feature creep, such as that proposed in this issue.

I'm not closing this immediately because I expect further discussion, and if/when some consensus is reached then it needs to be documented somewhere, but I'm no longer intending to change the behaviour in order to support #! py -x.y shebangs.

[timabroad]

It depends exactly which emulation you're in. Cygwin brings its own Python build, which should include a python3, and if you installed Python from the Windows Store then you'll get a python3 on PATH (though last I heard, Cygwin and anything based on it - which seems to be every Windows Bash implementation - isn't capable of launching Store apps, so that's not much help). The environment is different enough and far enough out of our control that we don't support it, though we try not to do anything to deliberately break it.

I'm using Git for Windows (which I think is built on top of MSYS2). It apparently doesn't come with a /usr/bin/python by default.

Personally for this use case, I'm not too fussed about py foo.py doing anything fancy with the shebang line at all. But if there's a valid Python install I'd rather it ran something rather than failing. That is, if I do have something weird or fancy in the shebang, it's unexpected that py foo.py can fail to even lookup the interpreter. So maybe if py is unable to handle the shebang correctly it should just fall back to whatever processing it would normally do to locate the version of python to run, and prefer running something rather than nothing? (For example, in the case above, the shebang makes it look like it it's looking for an interpreter called py in the CWD, but there isn't such an executable, so fall back to whatever default python install it can find)

Having said that, I'm not actually sure how/why my shebang was working with ./foo.py. From what I can tell, with a shebang of #!py, bash only looks in the current working directory (so you get Unable to create process using 'C:\test\py ./foo.py': The system cannot find the file specified.) but if you use #!py -3 suddenly it does a PATH search and works - not sure if this is documented behaviour or a bug in bash 🤷‍♂️.

[zooba]

Git Bash for Windows is about the most useless version of Bash, unfortunately, simply because it doesn't come with the full environment around it. As I said, we try not to actively break it, but there's very little we can do to make it work.

I suspect that when you ./foo.py in Git Bash, it's processing the shebang line itself, which will follow its own rules. Most likely, it just does a command line replacement and re-runs it, which means you'll get a normal CreateProcess search.¹ Because py.exe's handling of a bare command doesn't work like this, it will not get the same result.

So what likely happens is that the #!py shebang is used to launch py.exe foo.py which then reprocesses the shebang. But #!py -3 shebang (when launched from Bash) launches py.exe -3 foo.py which then ignores the shebang due to the -3, and so succeeds.

Probably the best fix here is for Get Bash to detect the /usr/bin/env python* pattern and remap it to py.exe itself. I'm not sure if that's possible to configure or if it'd need to be an upstream feature, but fundamentally the issue belongs to them.

Alternatively you could configure a command in py.ini that redirects a py shebang to a Python executable, which will override the script directory search we do for unknown commands. This appears to also be undocumented, unfortunately, but adding a [commands] section with py=<path> ought to do it.

Footnotes

1. Searches the application directory - bash.exe in this case - first, then PATH, with a few added complexities that depend on your system configuration. ↩

[timabroad]

I suspect that when you ./foo.py in Git Bash, it's processing the shebang line itself, which will follow its own rules. Most likely, it just does a command line replacement and re-runs it, which means you'll get a normal CreateProcess search. Because py.exe's handling of a bare command doesn't work like this, it will not get the same result.

So what likely happens is that the #!py shebang is used to launch py.exe foo.py which then reprocesses the shebang. But #!py -3 shebang (when launched from Bash) launches py.exe -3 foo.py which then ignores the shebang due to the -3, and so succeeds.

Oof! Yeah that sounds plausible and would explain the behaviour. Switching to #!/usr/bin/env py seems to be the simplest fix for me in this case as I don't need to worry about Linux support or anything. Thanks for the help 😃.