py.exe launcher ignores [commands] from py.ini

[cwalther]

Bug report

This is a regression between Python 3.11.0 and 3.11.1. Since 46a3cf4 (88297e2, #98692), py.exe no longer observes the “custom commands” mapping under [commands] in py.ini defining additional “virtual commands” (unless the virtual command starts with the same prefix as one of the four predefined virtual commands), but directly tries to launch the virtual command as an executable.

Steps to reproduce:

%WINDIR%\py.ini:

[commands]
/opt/something/bin/my-python2=C:\something\python27\python.exe

test.py:

#!/opt/something/bin/my-python2
import sys
print('hello from', sys.executable)

%WINDIR%\py.exe test.py

Expected result (and actual up to 3.11.0):

('hello from', 'C:\\something\\python27\\python.exe')

Actual result:

Unable to create process using 'C:\opt\something\bin\my-python2  test.py': The system cannot find the file specified.

I seem to be able to fix this as follows, which satisfies the existing tests, however this code is such a complex tangle of special cases that I have no idea whether it is the right thing to do. (The idea is that the loop over shebangTemplates should always find exactly one match, which was previously (before the regression) ensured by the empty template, so that _findCommand() is always called. Checking for tmpl != &shebangTemplates[0] is needed to satisfy test_recursive_search_path, however it might exclude too much – maybe searchPath() should instead report explicitly that it skipped a recursive call.)

diff --git a/PC/launcher2.c b/PC/launcher2.c
index 9b3db04aa4..ad313c10f3 100644
--- a/PC/launcher2.c
+++ b/PC/launcher2.c
@@ -1001,19 +1001,13 @@ checkShebang(SearchInfo *search)
         L"/usr/bin/env ",
         L"/usr/bin/",
         L"/usr/local/bin/",
-        L"python",
+        L"",
         NULL
     };
 
     for (const wchar_t **tmpl = shebangTemplates; *tmpl; ++tmpl) {
         if (_shebangStartsWith(shebang, shebangLength, *tmpl, &command)) {
             commandLength = 0;
-            // Normally "python" is the start of the command, but we also need it
-            // as a shebang prefix for back-compat. We move the command marker back
-            // if we match on that one.
-            if (0 == wcscmp(*tmpl, L"python")) {
-                command -= 6;
-            }
             while (command[commandLength] && !isspace(command[commandLength])) {
                 commandLength += 1;
             }
@@ -1052,18 +1046,20 @@ checkShebang(SearchInfo *search)
                     debug(L"# Treating shebang command '%.*s' as 'py'\n",
                         commandLength, command);
                 }
+            } else if (tmpl != &shebangTemplates[0]) {
+                // Unrecognised commands are joined to the script's directory and treated
+                // as the executable path
+                return _useShebangAsExecutable(search, shebang, shebangLength);
             } else {
                 debug(L"# Found shebang command but could not execute it: %.*s\n",
                     commandLength, command);
             }
             // search is done by this point
-            return 0;
+            break;
         }
     }
 
-    // Unrecognised commands are joined to the script's directory and treated
-    // as the executable path
-    return _useShebangAsExecutable(search, shebang, shebangLength);
+    return 0;
 }
 
 

Your environment

• CPython versions tested on: 3.9, 3.10, 3.11.0 (good), 3.11.1 (bad)
• Operating system and architecture: Windows 10 Pro 10.0.19043.2006 AMD64

Linked PRs

• gh-100247: Fix py.exe launcher not using entire shebang command for finding custom commands #100944
• [3.11] gh-100247: Fix py.exe launcher not using entire shebang command for finding custom commands (GH-100944) #101012
• gh-100247: Improve documentation for custom shebang commands in py.exe launcher #101083
• [3.11] gh-100247: Improve documentation for custom shebang commands in py.exe launcher (GH-101083) #101084

[cwalther]

Thinking about it some more, I question whether the call to _findCommand() should be inside the loop over shebangTemplates at all. That means that with a configuration of

[commands]
something/apython=C:\something\a\python.exe
/usr/bin/something/apython=C:\something\b\python.exe

a script with shebang line #!/usr/bin/something/apython will match the first line and be executed with a\python.exe. I see nothing of that sort specified in PEP 397. I would expect it to match the second line and choose b\python.exe. To me, the order of bullet points in Shebang line parsing says that _findCommand should come before before checking for virtual commands with one of the four known prefixes and should consider the entire command.

(However it seems that this has already worked this way in the 3.9 and 3.10 launcher.)

[zooba]

I think what probably needs to happen here (writing my logic down so I can double check once this headache is over) is that the templates should require python (i.e. /usr/bin/python..., not /usr/bin/*), and then the fallback should check for custom commands before assuming it's an executable.

I agree that the original PEP suggests the custom command should be checked first taking the entire first argument from the shebang, but as pointed out, that's not how it worked in the past. Implementation wins over the original propsal here (since the docs don't even mention this feature).

[cwalther]

Thanks! I’ll give it a test when I’m back at my work Windows desktop, probably some time next week.

This bit worries me a little, though:

The name of the command must be a single argument (no spaces),
and the value substituted is the full path to the executable (no arguments
may be added).

If that is true, then there may still be a regression compared to 3.11.0. The actual example we are using in our shipping product to make sure our scripts launch with our own Python installations on both Linux and Windows (which worked with the 3.9 launcher we ship but broke when users installed a 3.11.1 launcher) is

#!/opt/indel/bin/indel-python2 -u

with py.ini

[commands]
/opt/indel/bin/indel-python2=C:\indel\share\python\python27\python.exe
/opt/indel/bin/indel-pythonw2=C:\indel\share\python\python27\pythonw.exe
/opt/indel/bin/indel-python3=C:\indel\share\python\python3\amd64\python.exe
/opt/indel/bin/indel-pythonw3=C:\indel\share\python\python3\amd64\pythonw.exe
/opt/indel/bin/indel-python=C:\indel\share\python\python3\amd64\python.exe
/opt/indel/bin/indel-pythonw=C:\indel\share\python\python3\amd64\pythonw.exe

where on Linux /opt/indel/bin/indel-python* are symlinks to the appropriate interpreter while the Windows paths are the actual installation locations.

(I’m not 100% certain we really need the -u, but it could be because some scripts relay output from other processes that may include progress indicators that should not be line-buffered.)

[zooba]

Those are all fine. What it means is that you can't use /opt/indel/bin/indel-python2 -u as the command name - it'll stop looking at the first space.

Also, you can't use C:\indel\share\python\python27\python.exe -u as the right-hand side of the setting, because the whole thing is passed as the executable name, and is quoted when put into the command line.

[cwalther]

Ah, OK. That makes sense. “name of the command” refers to the the py.ini entry, not to the shebang line.

[cwalther]

I have tested it now and it works perfectly for our purposes. Thanks again!

It even fixes the paradoxical behavior I mentioned in #100247 (comment), which makes me happy.