build: better support for python3 systems #14737

bnoordhuis · 2017-08-10T12:24:00Z

Improve support for systems where python is actually python3.

Not all systems have a python2 binary, so simply updating the shebang
won't work.

What we can do is apply some cleverness: start life as a shell script,
locate the python binary, then re-execute the script but this time as
python code.

Special care is taken to ensure that spaces in arguments are passed on
verbatim.

This would need to be applied to other python scripts as well but I figured
I'd start with a single script in case everyone hates the approach.

CI: https://ci.nodejs.org/job/node-test-pull-request/9594/

addaleax

I’m okay with this.

gibfahn

Couple of suggestions

_{Also branch name should have been I-shpy-with-my-little-eye-something-beginning-with-python2}

gibfahn · 2017-08-12T22:58:29Z

configure

@@ -1,4 +1,14 @@
-#!/usr/bin/env python
+#!/bin/sh


This will stop syntax highlighting from working. Not the end of the world I guess.

gibfahn · 2017-08-12T23:00:37Z

configure

+which python2 >/dev/null && exec python2 "$0" "$@"
+exec python "$0" "$@"
+''' "$0" "$@"
+]



Maybe add a del _ or _ = None so that the _ variable doesn't propagate through the script. I doubt it'll conflict with anything (or cause a performance hit), but it would at least clarify that the python _ variable creation is a byproduct of the shell script.

I'll do that if you insist but my thinking was that this snippet will be copy/pasted across a number of files and should therefore be as succinct as possible.

If you're copy pasting an extra line doesn't matter right?

I find it clearer, so yes please.

gibfahn · 2017-08-12T23:05:22Z

configure

+# Locate python2 interpreter and re-execute the script.  Note that the
+# mix of single and double quotes is intentional, as is the fact that
+# the ] goes on a new line.
+_=[ 'exec' '/bin/sh' '-c' '''


Is changing $_ definitely okay in all shells? I know it's normally not user-modifiable, but IDK about csh and other arcane shells. I checked zsh, bash, and dash.

/bin/sh is always a Bourne-compatible shell, so it should be okay.

gibfahn · 2017-08-12T23:05:49Z

configure

+# the ] goes on a new line.
+_=[ 'exec' '/bin/sh' '-c' '''
+which python2.7 >/dev/null && exec python2.7 "$0" "$@"
+which python2 >/dev/null && exec python2 "$0" "$@"


Maybe check for python2.6 as well?

So, I deliberately didn't add python2.6 for two reasons:

I wasn't sure if it should go before or after python2, and

We supported python2.6 because of centos 5 but since we no longer support that platform...

Okay, I guess dropping python2.6 should be a separate PR.

gibfahn · 2017-08-12T23:13:32Z

Ben this is witchcraft, did you just come up with this? I can't get over how many python specific things it relies on.

bnoordhuis · 2017-08-14T09:53:15Z

I got the idea from a perl script. :-) It's a lot easier in perl though because just about any shell syntax is also valid perl.

Improve support for systems where `python` is actually `python3`. Not all systems have a `python2` binary, so simply updating the shebang won't work. What we can do is apply some cleverness: start life as a shell script, locate the python binary, then re-execute the script but this time as python code. Special care is taken to ensure that spaces in arguments are passed on verbatim.

bnoordhuis · 2017-08-17T12:05:54Z

Added a del statement. New CI for good measure: https://ci.nodejs.org/job/node-test-pull-request/9706/

jasnell · 2017-08-23T22:01:34Z

New CI: https://ci.nodejs.org/job/node-test-pull-request/9806/

Improve support for systems where `python` is actually `python3`. Not all systems have a `python2` binary, so simply updating the shebang won't work. What we can do is apply some cleverness: start life as a shell script, locate the python binary, then re-execute the script but this time as python code. Special care is taken to ensure that spaces in arguments are passed on verbatim. PR-URL: #14737 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>

jasnell · 2017-08-23T23:33:33Z

Landed in c6da5c8

Improve support for systems where `python` is actually `python3`. Not all systems have a `python2` binary, so simply updating the shebang won't work. What we can do is apply some cleverness: start life as a shell script, locate the python binary, then re-execute the script but this time as python code. Special care is taken to ensure that spaces in arguments are passed on verbatim. PR-URL: nodejs/node#14737 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>

Improve support for systems where `python` is actually `python3`. Not all systems have a `python2` binary, so simply updating the shebang won't work. What we can do is apply some cleverness: start life as a shell script, locate the python binary, then re-execute the script but this time as python code. Special care is taken to ensure that spaces in arguments are passed on verbatim. PR-URL: #14737 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>

MylesBorins · 2017-09-20T17:33:07Z

I've landed on v6.x. Please let me know if it should be backed out

c9n · 2020-07-19T08:08:18Z

configure

+exec python "$0" "$@"
+''' "$0" "$@"
+]
+del _



@bnoordhuis I'm not very clear how it works. Could you explain a bit more about this block for me? Thanks.

_=[ 'exec' '/bin/sh' '-c' ''' which python2.7 >/dev/null && exec python2.7 "$0" "$@" which python2 >/dev/null && exec python2 "$0" "$@" exec python "$0" "$@" ''' "$0" "$@" ] del _

@c9n The _=[ ... ] is first evaluated as shell script that re-executes the whole script as python code when it finds the appropriate interpreter. As python code, it's effectively a no-op: it assigns to _ an array of strings that isn't otherwise used.

@bnoordhuis Got it! Thank you so much.

nodejs-github-bot added the build Issues and PRs related to build files or the CI. label Aug 10, 2017

vsemozhetbyt added the python PRs and issues that require attention from people who are familiar with Python. label Aug 10, 2017

addaleax approved these changes Aug 10, 2017

View reviewed changes

danbev approved these changes Aug 10, 2017

View reviewed changes

cjihrig approved these changes Aug 10, 2017

View reviewed changes

jasnell approved these changes Aug 10, 2017

View reviewed changes

gibfahn reviewed Aug 12, 2017

View reviewed changes

bnoordhuis force-pushed the shpy branch from d912d25 to e1a700d Compare August 17, 2017 12:05

gibfahn approved these changes Aug 17, 2017

View reviewed changes

gibfahn mentioned this pull request Aug 19, 2017

Should intelligently try to detect Python 2.7 nodejs/node-gyp#1266

Closed

jasnell closed this Aug 23, 2017

MylesBorins mentioned this pull request Sep 10, 2017

v8.5.0 proposal #15308

Merged

MylesBorins added the land-on-v6.x label Sep 20, 2017

MylesBorins mentioned this pull request Sep 20, 2017

v6.11.4 proposal #15506

Merged

gireeshpunathil mentioned this pull request Oct 17, 2017

build: use bin path override if no python is found in PATH #16241

Closed

2 tasks

richardlau mentioned this pull request Aug 16, 2018

Possible python version bug, during build on MacOS High Sierra #22353

Closed

c9n reviewed Jul 19, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: better support for python3 systems #14737

build: better support for python3 systems #14737

bnoordhuis commented Aug 10, 2017 •

edited

Loading

addaleax left a comment

gibfahn left a comment •

edited

Loading

gibfahn Aug 12, 2017

gibfahn Aug 12, 2017

bnoordhuis Aug 14, 2017

gibfahn Aug 14, 2017

gibfahn Aug 12, 2017 •

edited

Loading

bnoordhuis Aug 14, 2017

gibfahn Aug 12, 2017

bnoordhuis Aug 14, 2017

gibfahn Aug 14, 2017

gibfahn commented Aug 12, 2017

bnoordhuis commented Aug 14, 2017

bnoordhuis commented Aug 17, 2017

jasnell commented Aug 23, 2017

jasnell commented Aug 23, 2017

MylesBorins commented Sep 20, 2017

c9n Jul 19, 2020 •

edited

Loading

bnoordhuis Jul 19, 2020

c9n Jul 19, 2020

build: better support for python3 systems #14737

build: better support for python3 systems #14737

Conversation

bnoordhuis commented Aug 10, 2017 • edited Loading

addaleax left a comment

Choose a reason for hiding this comment

gibfahn left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gibfahn Aug 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gibfahn commented Aug 12, 2017

bnoordhuis commented Aug 14, 2017

bnoordhuis commented Aug 17, 2017

jasnell commented Aug 23, 2017

jasnell commented Aug 23, 2017

MylesBorins commented Sep 20, 2017

c9n Jul 19, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnoordhuis commented Aug 10, 2017 •

edited

Loading

gibfahn left a comment •

edited

Loading

gibfahn Aug 12, 2017 •

edited

Loading

c9n Jul 19, 2020 •

edited

Loading